scrapers

scrapers copied to clipboard

Error: ``` \ScraperSetup.py", line 773, in create_button_pressed for i in range(len(lines_to_change)): TypeError: object of type 'NoneType' has no len() ```

CaptainStabs

Show status in scrapers table if it fails

1

Per Stabs' comment below, we should have clone of the Datasets db for Scrapers to use. Scrapers can update the db, which can in turn be synced to DoltHub.

josh-chamberlain

Wire up Scraping app CI to the Scrapers table

1

https://www.dolthub.com/repositories/pdap/datasets/data/master/scrapers As a scraper volunteer, I need to update this scrapers table using a variety of local utilities. It should be updated by the utility @CaptainStabs made, or the [intake...

josh-chamberlain

I finished CA (except for San Francisco), and now we need to finish the rest!

CaptainStabs

Update the [contribution](https://github.com/Police-Data-Accessibility-Project/PDAP-Scrapers/tree/v0.0.5#how-to-contribute) section of the readme to point to the gui

CaptainStabs

josh-chamberlain

Aurora, CO scraper requested

1

Discord user Steel [requested information](https://discord.com/channels/828274060034965575/849817327591096392/890929530351546408) from Aurora Colorado. https://www.dolthub.com/repositories/pdap/datasets/query/master?q=SELECT+*%0AFROM+%60agencies%60%0Awhere+%60city%60+like+%22Aurora%22&active=Tables agency id `b577cf2d087741e88e8851e1fc48baee` in our system, but no current datasets.

josh-chamberlain

Change wording to mean lowest path *that still leads to the police page* 

CaptainStabs

Part of https://github.com/Police-Data-Accessibility-Project/PDAP-Scrapers/issues/80 When a Scraper is run and an Extraction is submitted successfully, we should be dropping a URL / path into the [data_intake database](https://www.dolthub.com/repositories/pdap/data-intake).

josh-chamberlain

Set a run interval based on the dataset

2

josh-chamberlain