scrapers icon indicating copy to clipboard operation
scrapers copied to clipboard

Show status in scrapers table if it fails

Open josh-chamberlain opened this issue 3 years ago • 1 comments

Per Stabs' comment below, we should have clone of the Datasets db for Scrapers to use. Scrapers can update the db, which can in turn be synced to DoltHub.

josh-chamberlain avatar Aug 07 '21 23:08 josh-chamberlain

This one might be difficult to do depending on how we plan on using the scrapers We can't package credentials to our dolthub with the scrapers, so for them to update we will either need to run them from our own server or anyone who runs them will have to either be verified or have to login to dolthub and have it create a PR on it's own (automatically) And while it is possible to automate PR creation (see https://www.dolthub.com/repositories/dolthub/menus/pulls), that in itself isn't perfect as we don't want to spam dolthub with PRs

CaptainStabs avatar Nov 18 '21 16:11 CaptainStabs

outdated strategy

josh-chamberlain avatar Apr 06 '23 16:04 josh-chamberlain