Using Python to Check List of URLs

Recently at my job I had a need to check a long list of URLs to see if they were valid.  It was a simple list of links to images to make sure all of the images that a client had uploaded were going to work.  I didn't want to do them by hand, so I created this little Python script to do it for me.

Basically, all it does is to open a list of URLs and then use requests to submit that URL.  It then gets the response back.  If I get a 404, which means the file is not there, then it puts that URL in a list and moves on.  

Once the script is finished it puts the list of bad URLs into a file so the bad links can be shared with the client.

I'm not sure if anyone else will find this useful, but I've used it several times lately and wanted to share it.

The code is as follows:

import requests
import pandas
flaturl = []
print('importing file')
urllist = pandas.read_csv('urls.csv')
urllist2 = urllist.values.tolist()
print('flattening file')
for sublist in urllist2:
for item in sublist:
flaturl.append(item)
print('sorting list')
flaturl.sort(reverse = True)
brokenlist = []
print('checking urls')
for l in flaturl:
r = requests.get(l)
c = r.status_code
print(l, ' ', c)
if c != 200:
brokenlist.append(l)
#print(brokenlist)
df = pandas.DataFrame(brokenlist)
df.to_csv('BrokenURLs.csv', index=False)

Let me know if you find this useful. 

Impose Constraints to be MORE Creative

What if I asked you to tell me a story, write a song, or draw a picture?  Would you be able to do so without any direction or prompts?  What should the story contain? What colors should you use in the picture?  Would you be stumped and struggling to have a good idea to run with?

What if I instead told you to tell me a story about a space ship?  What if the song should be about your first kiss?  How about drawing a picture of a rose?  Would these constraints make it easier for you to come up with a creative idea?  If you are like most people the answer is "yes".

Let's look at another example of how constraints can aid in creativity.  Let's say you are an architect and I approach you and say "Design a house for me."  Where would you start?  Would you design a split-level ranch, a modern, sleek master piece, or a Victorian revival style home?  Would it have one story or two?  How many bathrooms?  How many bathrooms?  These questions can go on and on.

Now, suppose I came to you and said that I want a 5 bedroom, 4 bath, with a large study, a bonus room, and a large porch.  Working with these constraints you can begin to design a house.

What does this all mean to you?  Well, in short, don't see constraints as limits.  See them as a means of freeing you up to focus on what is important to your project.  While constraints eliminate a large selection of options that are initially available, doing so allows you to focus on what you can actually work with.  It is within these constraints that your brain is free to create and be innovative.

So, go out and embrace those constraints that you once saw as limits and use them to become more creative that you thought was possible before.

Using Evernote for Photography Scouting [Evernote]

I am by no means a professional photographer, but I do love taking pictures.  One of my favorite things to do is to find some really cool places to go and take pictures at different times of day.  I'm always on the lookout for a cool place to stop and take pictures.  However, sometimes I simply can't stop long enough, whether due to having to get to work, or having an impatient child in the vehicle, or I just don't have my camera with me.  In those cases, I really want to be able to remember that location, but the odds of me remembering that location long enough to get home to write it down, or finding the note that I write in the vehicle, is pretty slim.

Enter Evernote.  

Evernote is one of my favorite apps on my iPhone.  I use it to find notes that I save, take pictures of business cards for future reference, recipes, and so much more.  It's kind of like a second brain for me.  In this case it serves as my personal photo scout.  What I do is to open the app on my phone, go to a notebook, in my case it's called "Pic Scouting".  From there I chose the camera icon and Evernote launches it's built-in camera function, and take a picture of whatever interesting thing I see.  Now, this is not a full featured camera, and it's not an ideal way of taking photos that you want to be very high quality.  However, what it does have going for it is that when you take a picture is that it will instantly upload the image into your chosen notebook.  The real power of this is that Evernote will apply the GPS coordinates to that photo.  

What this does for me is that a week from now when I have time to get out and go take some photos, all I have to do is to open my "Pic Scouting"notebook in Evernote, find the location that I'm most interested, or fits into the amount of time I have, pull out the address, put it in some map app on my phone and take off.

Are you using Evernote for some similar type of application?  I'd like to hear what you are doing with it.