scrapers icon indicating copy to clipboard operation
scrapers copied to clipboard

Wire up Scraping app CI to the Scrapers table

Open josh-chamberlain opened this issue 3 years ago • 1 comments

https://www.dolthub.com/repositories/pdap/datasets/data/master/scrapers

As a scraper volunteer, I need to update this scrapers table using a variety of local utilities.

It should be updated by the utility @CaptainStabs made, or the intake utilities, or similar.

For more context, discussion: https://discord.com/channels/828274060034965575/828283062827221083/860953464527388713

josh-chamberlain avatar Jul 03 '21 18:07 josh-chamberlain

Programmatically creating PRs on dolt is all done through GraphQL queries. I'm not entirely sure how we would provide an easy way for volunteers to feed the creation utility their cookie. The other issue with creating a scraper ID in dolt via the utility is that it's not accessible for someone who just wants to create a single scraper (have to download more software). This may be more of a task for us to do automatically behind the scenes (git diff to find the new folder(s))?, generate a guuid, perhaps add author(s) to the scraper table, and then insert it.

CaptainStabs avatar Aug 09 '21 20:08 CaptainStabs

outdated, not storing stuff in Dolt

josh-chamberlain avatar Apr 06 '23 16:04 josh-chamberlain