deadoralive

deadoralive copied to clipboard

Add content-type support

1

Allow the client site to (optionally) specify an expected Content-Type with each link. We can then assert that the Content-Type header is correct. We could also support parsing for certain...

seanh

Fail for really large responses

2

I'm not sure what currently happens if the server sends a really large response. We should probably add a "too big" failed link check type. See here for details on...

seanh

We had some problems with some sites only talking when using particular versions of SSL. Some like v23 and some like v3. So we retry: ``` from requests_ssl import SSLv3Adapter...

davidread

You might want to consider how to categorize when a 403 is returned. We have some paid-for data listed which is behind an HTTP Basic Auth password. You would be...

davidread

Handle different link check failures differently

1

We currently just try to catch any exception that we think `requests` might throw and put together a failed link check report (status and reason) in a generic way. It...

seanh

Add test for URLs with non-ASCII characters in them

1

`requests.get(url=u"http://➡.ws/mobify")`

seanh

Test all sorts of HTTP responses

seanh

Set the `Accept-Content: gzip` header. `requests` should be able to decompress it.

seanh

For `https` URLs pass `vefiry=True` to `requests.get()` and catch `requests.exceptions.SSLError`

seanh

Retry failed requests a configurable number of times before reporting a broken link to the client site. See http://www.mobify.com/blog/http-requests-are-hard/ for how to do this with requests.

seanh