Harvester Timed Out

[bwagerson]

Harvester timed out at about 677,989 out of 1,500,000 items while trying to harvest all of Arxiv.org, is there a way to pick the harvest back up where it timed out? Instead of starting at the beginning?

[bloomonkey]

Hi. See #22 for some conversation around this

[ericywl]

Sorry, but is the definitive answer to provide the resumption token? How would we get that from oai-harvest?

[bloomonkey]

Yes, the resumptionToken is the only mechanism in OAI-PMH for resuming a previous harvesting run. I'm not sure how you'd get access to the necessary token though, as it's probably only used internal to oaiharvest to retrieve the next chunk 🤔 . Maybe it could store the token in a local file called e.g. .resumptionToken and use this as a default for the -r option if not value is provided...