`forget`: Add option to automatically merge snapshots

[enboig]

trafficstars

When deleting snapshots, I would link to keep the latest version of a file that has been deleted. An example is:

• I created a file 2nd of month, worked on it until 25th, and then deleted it. Keeping monthly files will forget about this file, so I would like to keep a snapshot for a day "between previous and next" with last version of files created/updated/deleted between "previous" and "next". I know this snapshot would not be consistent (a git repo would fail...), but when working with documents it could be important. I don't know if this could be related to rustic-rs/rustic#43. Thanks

[aawsome]

Thanks for proposing this issue!

I think I partly understood your intention, but I'm not perfectly sure if I did get all details. Can you clarify your request a bit?

Are you thinking of

• keeping one particular (known) file, or
• keeping all files

at least in one snapshot?

And what do you think rustic should do to preserve the file:

• modifying snapshots such that only the file(s) removed in follow-up snapshots are contained and then keeping the modified snapshots?
• keeping snapshots which have the "latest" version of the file(s)?

[aawsome]

Ok, I think I might have understood what you mean (or at least I can imagine an algorithm which may suit your needs):

• use all snapshots within a given time range (+ maybe fulfilling other filter criterias)
• merge all snapshots with the following merging strategy:
  • if a file exists in a snapshot, but not in another, put it in the merged snapshot
  • if a file exists in many snapshots in different versions, use the one of the latest snapshot
• mark the generated snapshot with a "no delete" mark (this is already implemented)

Is this what you are asking for?

[enboig]

Yes, it is.

I like the "no delete"; but I would tag it with a "dirty", "inconsistent", "frankenstein" tag

[enboig]

And the algorithm should be:

• when forgetting a snapshot:
  • if there is a previous snapshot which is already "inconsistent", copy my files over it adding and/or updating existing ones.
  • if not, create a new one with the previous "consistent", and copy my files over it adding and/or updating existing ones.

This way you would have always a "intersnapshot" containing the last version of every files, including the ones created and deleted between the full snapshots

[aawsome]

Marking this as duplicate because I think rustic-rs/rustic#43 will solve this.

[enboig]

Well... I would say they are related; but I was asking for an option when "forgetting" snapshots which automatically take care of merging snapshots which are being forgotten.

[aawsome]

Well... I would say they are related; but I was asking for an option when "forgetting" snapshots which automatically take care of merging snapshots which are being forgotten.

I was actually thinking about "merging" snapshots with an option to remove (a.k.a forget) the snapshots which are merged. So IMO your request can be satisfied with the new merge command.

That said, the filtering of the snapshots-to-be-merged can be a bit tricky, but here I have two proposals:

• use the --filter-* options known by forget and maybe add more to satisfy the use case
• extend the forget or tag command to set specific tag(s) on snapshots which would be forgotten by forget. Then a run of merge --filter-tags <TAG> would do the trick.

[enboig]

I like the idea of extending forget with something like --keep-files-last-version which merge snapshots to be forgotten and tag them someway so user (and rustic itself) identify them. But I would talk to restic developers first, keeping both tools as much similar as possible would be nice.

[aawsome]

@enboig The merge command has been implemented (see rustic-rs/rustic#411) and is available in the latest beta builds.

You can use rustic merge --delete <snap1> <snap2> <snap3>... to merge and remove a list of snapshots. Moreover, you can use rustic merge --delete --filter-* to merge and remove all snapshots which satisfy a filter.

I hope this fits your usecase, but don't heasitate to write if something is missing...

[enboig]

Thanks, I will take a look at it. Can an "already taken" snapshot be tagged with extra tags?

[aawsome]

Can an "already taken" snapshot be tagged with extra tags?

Sure. There is the rustic tag command to add/remove/set tags. Note that this changes the snapshot id.

[enboig]

When merging snapshots, time of new snapshot is set to now; could it be set to "newest date of merged snapshots"?

[aawsome]

Yes, you are right. This is a better fit for the timestamp - especially as for identical files, also the latest version is taken. I'll change this.

[enboig]

and when there are identical snapshots, there appear a (+2) next to it; could it appear a (M) next to a merged snapshot? or should I tag them manually?

[enboig]

And after some testing, the snapshot order in terminal isn't taken into account when merging; they are always merged keeping newest version of each file (which I think is the best option). Am I right?

[aawsome]

and when there are identical snapshots, there appear a (+2) next to it; could it appear a (M) next to a merged snapshot? or should I tag them manually?

Sorry, I didn't get what you are asking. Can you try to rewrite which behavior you are expecting? Note that the (+2) is only a output-format thing. You can always see all snapshots if you use the --all option or when you output the snapshots in json format.

[aawsome]

And after some testing, the snapshot order in terminal isn't taken into account when merging; they are always merged keeping newest version of each file (which I think is the best option). Am I right?

Right. I think the order of snapshots shouldn't play any role. Currently the newest version (i.e. latest modification timestamp) is taken if a file is contained in more than one snapshots. Please open an issue if you need other merging strategies!

[enboig]

Could it be any way of identify merged snapshots? What are "labels" used for? (I don't remember seeing them in restic)

[aawsome]

Could it be any way of identify merged snapshots? What are "labels" used for? (I don't remember seeing them in restic)

You can use a label and/or tags to identify them, e.g. rustic merge --label LABEL --tag TAG1,TAG2. restic doesn't support labels, which means that a snapshot with a label can be used in restic, but it is not displayed or can be used for anything.

Another possibility is to use a description (also not supported in restic) rustic merge --description "My description", but there is no way to e.g. filter for it and it is only shown by rustic snapshots --long or rustic snapshots --json.

BTW: rustic-rs/rustic#501 is merged and available in the latest beta build.

[enboig]

Returning to my initial idea... my objective is being able to rustic forget ... --keep-all-merged and go from

to

I know id of merged snapshot would change, the status of "merged snapshot" shouldn't be able to be modified by user. And

id	action
1	keep
2	forget
3	forget
4	keep
5	forget
6	keep

Should become

id	condition
1	normal
2'	merged
4	normal
5'	merged
6	normal

(status or condition of a snapshot (normal, merged, ...) should be set by rustic itself, not by user)

[aawsome]

Sorry for coming back lately to this topic. I know, that what you intend is not yet possible with the forget command, but maybe the -filter-fn option would work for you.

In your example, you could do something like

rustic merge --filter-fn '|sn| sn-time < "2023-03-17T15:58:30"' --delete --label Merged

To test out our --filter-fn funciotn, run it on the snapshots command and view the result...