Mark Jordan

Update: the new metadatamanipluator works, but it interferes with other metadatamanipulators. Now all I have to do is figure out why it's doing that and make it stop :face_with_head_bandage:

I don't think resumption tokens would work here since they are internal to the OAI harvester, and because they represent chuncks of records, usually 100 but I think that is...

@bondjimbond just to clarify, when you say "to resume downloading a large set if the process freezes", do you mean the OAI part of the harvest, or the downloading of...

Consulting the fine docs at https://github.com/MarcusBarnes/mik/wiki/Toolchain:-OAI-PMH-for-repositories-that-identify-resource-files-in-a-record-element I see that this toolchain already supports the SpecificSet fetcher manipulator. We could give that a try. It applies to the OAI harvesting, not...

> SpecificSet is good and helpful in limiting the download - but when a "set" contains 6000+ objects, the possibility of timeouts/crashes/etc is high. You can split up large sets...

> You would run MIK configured with this manipulator as many times as necessary until you got all your files. I say that on the assumption that it's OK to...

Agreed about nice to have. Such a filegetter manipulator may be applicable to other toolchains that get files from a remote URL, both existing and yet-to-be-imagined ones. Let me put...

A quick look at the two existing filegetter manipulators at https://github.com/MarcusBarnes/mik/tree/master/src/filegettermanipulators reminds me that they are additive - they return likely paths to files. The use case we are circling...

Pondering this while eating my lunch, I think we can base this fetcher manipulator on [SpecificSet](https://github.com/MarcusBarnes/mik/blob/master/src/fetchermanipulators/SpecificSet.php) pretty easily. All we need to do is replace the code inside its `getSpecificSet()`...

I can give this a first crack over the weekend. I think it should be pretty straight forward.