warn-scraper icon indicating copy to clipboard operation
warn-scraper copied to clipboard

utils.fetch_if_not_cached needs response return

Open stucka opened this issue 1 year ago • 0 comments

A number of scrapers are using utils.fetch_if_not_cached, which itself will flag errors but they don't get passed back to the main program. See handling of get_url as an example.

This disallows the use of response.raise_for_status() in the main code.

However, fetch_if_not_cached would not be able to return a response object if it doesn't need to download something; in this case, that response can be faked: https://stackoverflow.com/posts/23914464/revisions

... and thus allow for consistent code.

Existing scrapers using this function would need to be updated: ga (x2) or tn

stucka avatar Aug 29 '24 10:08 stucka