Suggestion: Timeout for url

[SaschaHornauer]

Hi,

I realized a lot of the URLs do take quite a long time to finally state they can not be reached. If you'd use urllib.urlopen instead of urlretrieve you can define a timeout. Like this:

try:
    request = urllib.urlopen(url, timeout=500)
    with open(os.path.join(output_folder, fname), 'wb') as f:
        f.write(request.read())		
except Exception as ex:
    print "Could not retrieve url: ", url, " | skipping" 
    print ex

I would write a pull request but I am using urllib2 and python2.7 and lack a bit of motivation to install python3 to test it adequately.

[akshaychawla]

Hey @SaschaHornauer sorry for the late reply. I remember debating whether to use urllib.urlopen or urlretrieve but I forgot why I chose urlretrieve. I'll make the changes you suggested by the end of the week. Thanks for the suggestion :smile: