warn-scraper icon indicating copy to clipboard operation
warn-scraper copied to clipboard

Build automated QA checks

Open stucka opened this issue 1 year ago • 1 comments

As @Kirkman found in #597 , a scraper can stop producing output without triggering an error in workflow.

While a few states keep WARN and non-WARN layoffs in the same database it's unlikely that many states would ever have a reduction in the number of incidents reported in the files getting scraped. So if a state moves from 283 reports to 123 reports or 0 reports that should get flagged. Simple row counts of CSVs compared to earlier snapshots would have caught Missouri problem.

Weekly Github Action built in warn-support repo, perhaps?

stucka avatar Jan 05 '24 23:01 stucka

Looks like how to approach this problem is being considered over in https://github.com/biglocalnews/warn-transformer/issues/236

chriszs avatar Mar 10 '24 11:03 chriszs