warn-scraper
warn-scraper copied to clipboard
utils.fetch_if_not_cached needs response return
A number of scrapers are using utils.fetch_if_not_cached, which itself will flag errors but they don't get passed back to the main program. See handling of get_url as an example.
This disallows the use of response.raise_for_status() in the main code.
However, fetch_if_not_cached would not be able to return a response object if it doesn't need to download something; in this case, that response can be faked: https://stackoverflow.com/posts/23914464/revisions
... and thus allow for consistent code.
Existing scrapers using this function would need to be updated: ga (x2) or tn