warn-scraper icon indicating copy to clipboard operation
warn-scraper copied to clipboard
verified publisher icon biglocalnews

Small DC scraper problems

Open stucka opened this issue 1 year ago • 2 comments

The scraper assumes a link for 2014 is listed but echoes a different year. For 2024's index, at least, that is not the case; no link to 2014 is offered, and therefore the scraper will not automatically try to download 2014.

Scraper grabs old HTML even when there's no update, e.g., in 2024 maybe we want to update 2024 and 2023, but we don't need to grab 2019 again. A small optimization might be to try to hit older URLs with utils.fetch_if_not_cached and then read from it the files with cache.

stucka avatar Mar 25 '24 14:03 stucka

I'm sure there was a reason to use this uuid thing instead of labeling the files with the years, but I'm not seeing it at the moment. =)

stucka avatar Mar 25 '24 14:03 stucka

Also should maybe nix the http:// prefix in the use of 2014.

stucka avatar Mar 25 '24 14:03 stucka