Explicit DNSError subclass for failed DNS lookups

[techtonik]

There is no cross-platform way to catch DNS lookup errors with requests. At least I see that error message on Windows https://stackoverflow.com/questions/40145631/precisely-catch-dns-error-with-python-requests looks different from Linux in https://github.com/kennethreitz/requests/issues/3550

Would be nice to get dedicated exception for this use case.

[Lukasa]

So this is somewhat reasonable, but made more opaque by the fact that urllib3 gets in the way here. urllib3 will basically always turn a connection setup problem like this one into a MaxRetriesError (because of its retry processing). That means for us to fire off an appropriate error in all cases means that we need to go and look at every possible exception that can fire.

I think, as a first step, you should propose that urllib3 raise special exceptions when it gets problems from getaddrinfo. From there, Requests can introspect and look for such an exception, which gives us a better shot at raising an appropriate error. And at the very least, you will then be able to dig in to the exceptions to pull out the original (as all of those exceptions are just wrappers).

[techtonik]

Reported above. Do you have access to OS X machine to test what requests.head('http://wowsucherror') returns there currently?

[Lukasa]

In 2.7.12:

requests.exceptions.ConnectionError: HTTPConnectionPool(host='wowsucherror', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<requests.packages.urllib3.connection.HTTPConnection object at 0x10e22ff10>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

[techtonik]

The hack so far:

import requests

def sitecheck(url):
    status = None
    message = ''
    try:
        resp = requests.head('http://' + url)
        status = str(resp.status_code)
    except requests.ConnectionError as exc:
        # filtering DNS lookup error from other connection errors
        # (until https://github.com/shazow/urllib3/issues/1003 is resolved)
        if type(exc.message) != requests.packages.urllib3.exceptions.MaxRetryError:
            raise
        reason = exc.message.reason    
        if type(reason) != requests.packages.urllib3.exceptions.NewConnectionError:
            raise
        if type(reason.message) != str:
            raise
        if ("[Errno 11001] getaddrinfo failed" in reason.message or     # Windows
            "[Errno -2] Name or service not known" in reason.message or # Linux
            "[Errno 8] nodename nor servname " in reason.message):      # OS X
            message = 'DNSLookupError'
        else:
            raise

    return url, status, message

print sitecheck('wowsucherror')
print sitecheck('google.com')

('wowsucherror', None, 'DNSLookupError')
('google.com', '302', '')

[dnx-seek]

I'm not sure what the status of this is a few years later... I found the above didn't work for me on current python, so came up with this. Note I'm only returning the OK status here.

def check_live(url):
    try:
        r = requests.get(url)
        live = r.ok
    except requests.ConnectionError as e:
        if 'MaxRetryError' not in str(e.args) or 'NewConnectionError' not in str(e.args):
            raise
        if "[Errno 8]" in str(e) or "[Errno 11001]" in str(e) or ["Errno -2"] in str(e):
            print('DNSLookupError')
            live = False
        else:
            raise
    except:
        raise
    return live

[gdubicki]

In case someone is wondering what is the status as of now: urllib3 v2.0 will have a separate exception for this, see https://github.com/urllib3/urllib3/commit/1831327b881880ed871f96f56a6977d360042e1b and https://github.com/urllib3/urllib3/commit/8a1ac9f8db13d0673188d542152253d6e133eec9.