Receiving empty response HTTP errors.

[Dracovian]

This is likely caused by the missing "Accept-Encoding: gzip" header from each request. Cloudflare returns this response header when it errors out: "Vary: Accept-Encoding".

The Vary header is used to determine whether to serve the page/file contents from cache. I haven't yet figured out if a missing cache is the culprit of this error, but I'm currently running the script again with the new header in place.

I'll return to this issue to state whether the additional header fixes the problem or not.

[EDIT]

Nope, still returning empty responses with the new header. I'll definitely have to keep a lookout for what could be causing this. In the meantime I might just handle downloads like this:

1. Grab all of the JSON contents and put them into an object (heavy memory usage).
2. Sift through the JSON contents for the contents we're looking for (slow).
3. Download the contents we find, hoping that they didn't expire before we could get to them.

I'll probably think of something better before I merge this branch anyhow.