civic-scraper icon indicating copy to clipboard operation
civic-scraper copied to clipboard

Standardize primegov.site.Site.scrape

Open zstumgoren opened this issue 3 years ago • 0 comments

Site.scrape method should be updated to match the features supported in CivicPlus and Legistar. Specifically, we should attempt to implement the following features:

  • Standardize site class name to Site and import as PrimeGovSite (the pre-existing name in civic_scraper.platforms.__init__.py)
  • File download should be set to False by default
  • date-based filtering, with default set to current day
  • file type filtering for downloads, with Agendas and Minutes set to default
  • file size filtering for downloads

We should also attempt to implement the cache flag as in CivicPlus, although we didn't do so for Legistar since the underlying python-legistar framework would need to be refactored to support caching of source HTML.

zstumgoren avatar Jun 22 '22 20:06 zstumgoren