scrapers
scrapers copied to clipboard
Code relating to scraping public police data.
Police Data Accessibility Project Scrapers
This repo contains the data scrapers for Police Data Accessibility Project. Thank you for being here!
How to run a scraper
Right now, this requires some Python knowledge and patience. We're in the early stages: there's no automated scraper farm or fancy GUI yet.
- Install Python.
- Clone this repo.
- Find the scraper you wish to run. These are sorted geographically, so start by looking in
/USA/...
. - Run the
scraper.py
file with something likepython3 <scraper path>
depending on how you installed it.
Did it work?
If it worked, discuss your findings in our Discord. If it didn't, make an issue in this repo or reach out in Discord.
How to contribute
To write a scraper, start with CONTRIBUTING.md. Be sure to check out the /common folder!
For everything else, start with docs.pdap.io.
What data are we scraping?
The datasets listed here are our to-do list. If we should targeting a new data type, suggest it in Discord or make a DoltHub PR!
Resources
Potentially useful tools. If you find something useful, or if one of these is out of date, make a PR!
- https://www.scrapingbee.com/
- https://github.com/CJWorkbench/cjworkbench