scrapex
scrapex copied to clipboard
Set last-modified and etag headers on request
So target server can send 304 if applicable.
Also, keep a hash of the content. That way, when receiving 200 response, we can do a quick check for change for servers that don't support etag. This reduces number of results that need to be parsed.