scrapers

scrapers copied to clipboard

### Context Our file downloaders could use a bit of a rework. They seem overly complex and only able to support a few different file types; with various modules calling...

EvilDrPurple

The end goal: a magic button for any Scraper that says something like `Run Scraper Locally`. When this button is clicked, the user needs to do as little as possible...

josh-chamberlain

Create archives of raw responses for Data Sources automatically

1

The prototype: https://github.com/Police-Data-Accessibility-Project/automatic-archives Great writing about why this is important: https://github.com/pdapm/intro ## This issue is about the long term strategy: - PDAP will need its own storage holdings, which are...

josh-chamberlain

CKAN general scraper

9

#### Fixes * Part of https://github.com/Police-Data-Accessibility-Project/data-projects/issues/9 #### Description * Adds a general scraper for scraping from CKAN instances #### Docs * See README.md

EvilDrPurple

Addresses [this issue](https://github.com/Police-Data-Accessibility-Project/data-projects/issues/8). This is a preliminary PR introducing two primary tools: - `create_foia_data_db.py`, which creates a local SQLite database in order to efficiently search MuckRock's FOIA requests collection -...

eddie-m-m

This came in handy for an Allegheny County research project ### Fixes - none ### Description - uses beautifulsoup to scrape, prettify, and CSV-ify departments, and areas without departments in...

josh-chamberlain