Is the resume manifest really necessary?

[gatl]

If we store a JSON object with the list of all games at the top of the manifest, and then add one JSON object entry per game, we can always know where we left off and proceed. Say, "our collection is 54 games, I could read only 43, let me work on the missing ones".

If we want to start over, we could just delete the manifest file! I think it is a pretty clean solution.

[gatl]

Perhaps I misunderstood the purpose of the manifest file. I thought it was to keep a local copy of the games and files that could be retrieved, and the resume is used for when the creation of this list is interrupted. The code for storing the regular manifest and the resume manifest is the same, so I expected the output files to be the same too.

Could you perhaps talk a little about the resume manifest and what could go wrong without it, if you would not mind?

[Kalanyr]

The resume manifest is used to resume an interrupted manifest update (the type down by the update command ), it keeps tracks of the games that haven't been updated yet , so the processing can resume at the point it was interrupted.

It's format is the same as the real manifest but it's used as a skeleton, the info from the game available at the library page level is stored in the resume manifest if the game is marked as Updated (or is New (ie not in the manifest)), when the library scan is done, but before the individual game or it's sub-items are scanned.

[gatl]

Please consider the following (I can see if I find the time to send you this in code, to test on the current version of gogrepo).

A manifest file consists of two sections:

• The "overview" part with the list of games in the collection, and
• The "detailed" part with the files and info for each game in sequence (the entries are loaded one by one). We only need to ensure that both parts sort the games in the same, consistent way.

Creating or updating the manifest would be done as follows (possibly similar to what is being done):

1. Read the overview from the manifest if one exists;
2. Request from GOG an updated list of games, and sort it;
3. Start a new manifest, based on the updated overview;
4. For each game in this new list: a. If it is a new game get the data and add to the new manifest; b. If is a game flagged as updated, get the data, add to the new manifest, ignore the old game data; or c. Copy the old game data that is still good to the new manifest;

If we have enough RAM (a very likely situation) we could load the entire manifest to memory and use just one file. Here we can verify that the "detailed" list is not complete (the execution was interrupted) and use the "overview" portion to determine from which point to carry on.

I'm sure you have spent far longer on this thing. I'm hoping you will tell me what I am ignoring or dismissing.

[Kalanyr]

Sorry for the delay I've looked into this now , the issue is that there isn't an "overview" part currently, the manifest itself just has the detailed part.

The resume manifest does use a kind of "overview" sketch but that's because it was trivial to use the sketch from the first stage of the update command to create the resume list rather than doing some kind of transformation.

I couldn't change the manifest structure to radically at the time I implemented resume for backwards compatibility purposes.

I'll think about implementing something like this with the new format I'm going to introduce.

[gatl]

It's OK. My free time is also quite spotty. Seems like it will take you a while until you present your new version. I'll see if I can find some low hanging fruit to work on until then.