github-to-sqlite
github-to-sqlite copied to clipboard
dogsheep
Command for importing events
Eg from https://api.github.com/users/simonw/events
Docs here: https://developer.github.com/v3/activity/events/#list-events-performed-by-a-user
I should build the command to persist ETags and obey their polling guidelines:
Events are optimized for polling with the "ETag" header. If no new events have been triggered, you will see a "304 Not Modified" response, and your current rate limit will be untouched. There is also an "X-Poll-Interval" header that specifies how often (in seconds) you are allowed to poll. In times of high server load, the time may increase. Please obey the header.
Prototype:
pip install sqlite-utils paginate-json
paginate-json "https://api.github.com/users/simonw/events" | sqlite-utils insert /tmp/events.db events - --pk=id
Since events include payloads with full object representations in them (for issues, repos and more) running this command every few minutes may be all it takes to keep a constant copy of everything updated in a very rate-limit friendly manner (thanks to the ETags).