civic-scraper
civic-scraper copied to clipboard
Standardize primegov.site.Site.scrape
Site.scrape method should be updated to match the features supported in CivicPlus and Legistar. Specifically, we should attempt to implement the following features:
- Standardize site class name to
Siteand import asPrimeGovSite(the pre-existing name incivic_scraper.platforms.__init__.py) - File
downloadshould be set toFalseby default - date-based filtering, with default set to current day
- file type filtering for downloads, with Agendas and Minutes set to default
- file size filtering for downloads
We should also attempt to implement the cache flag as in CivicPlus, although we didn't do so for Legistar since the underlying python-legistar framework would need to be refactored to support caching of source HTML.