BorgBackup v2 conversion

[m3nu]

The upcoming version 2 will have a number of breaking changes to clean up more legacy features and code.

• [ ] borg transfer to transition 1.x repositories to the new 2.0 repo format. Ask user before this irreversible action
  • [ ] scp style repository syntax is gone in favor of ssh://. Convert existing URLs in DB?
• [ ] Handle adding v1 repos with borg v2 installed - and the other way around.
• [x] Specify repository with new -r flag instead of :: syntax:
  • [x] prune
  • [x] compact
  • [x] create
  • [x] check
  • [x] extract
  • [x] mount
  • [x] umount
  • [x] list
  • [x] info
  • [x] delete
  • [ ] ~~borg~~
• [x] Specify archive with new positional parameter for most Borg sub-commands instead of :: syntax.
  • [x] mount
  • [x] create
  • [x] extract
  • [x] list
  • [x] delete
  • [x] info
• [x] Specify archive with --glob-archives or -a flag with a glob matching multiple archives for certain Borg sub-commands, e.g. borg delete --glob-archives 'myarchive*' or `borg recreate --glob-archives 'myarchive*'.
  • [x] mount
  • [ ] ~~delete~~
  • [x] info
• [x] Replace use of --prefix with --glob-archives, e.g. --globarchives 'myprefix*'.
• [ ] ~~Modify GUI when using borg v2 to allow selecting multiple archives for the mount operation.~~
• [x] Available encryption algorithms changed.
• [x] Repository sub-commands have been split off into new commands:
  • [x] init -> rcreate
  • [x] list -> rlist
  • [x] info -> rinfo
  • [ ] ~~delete -> rdelete~~
• [ ] Other Borg sub-commands may have changed as well. Audit and update:
  • [ ] prune
  • [ ] compact
  • [ ] create
  • [ ] check
  • [ ] extract
  • [ ] export-tar
  • [ ] mount
  • [ ] umount
  • [ ] restore
  • [ ] list
  • [ ] info
  • [ ] borg
• [ ] borg rcreate --other-repo flag.
• [ ] borg compact --cleanup-commits flag has been removed.

Full changelog: https://borgbackup.readthedocs.io/en/2.0.0b1/changes.html#version-2-0-0b1-2022-08-08

Discussion tickets: https://github.com/borgbackup/borg/discussions/6952 and https://github.com/borgbackup/borg/discussions/6799.

[real-yfprojects]

While we can deal with smaller differences, Vorta will also make a breaking change to properly adjust. Probably as Vorta v1.x

If we only need minor changes, I wouldn't call them 'breaking' for vorta and would only increase the minor version number to v0.9. Though I find it difficult to project semver onto GUI development.

[sophie-h]

If we only need minor changes, I wouldn't call them 'breaking' for vorta

You have probably missed that Vorta will stop working with older borg repositories when switching to borg 2.0. (Or Vorta has to ship with borg 1.x and borg 2.0 and choose between both depending on the repository.)

[real-yfprojects]

You have probably missed that Vorta will stop working with older borg repositories when switching to borg 2.0.

I would oppose that since this does not match vorta's philosophy. Vorta has always supported a bunch of borg versions with different feature sets. This allows the user to use the borg version meeting his needs the most while still having the latest vorta GUI. This will be especially useful for borg 2.0 that does only support python3.9 or later while 3.7 is still supported and many linux machines cannot easily upgrade their python version.

(Or Vorta has to ship with borg 1.x and borg 2.0 and choose between both depending on the repository.)

This is not (easily) possible since vorta uses the system borg installation which is only shipped in case of the flatpak distribution. It makes sense to provide two flatpak bundles with different borg versions since I imagine that many flatpak users won't want to switch to borg 2.0 yet.

[m3nu]

Updated our task list by stealing and adjusting the good one Borgmatic made here.

We can probably support all Borg versions with the same Vorta version. So no hard version increase.

[real-yfprojects]

I guess this should have quite the high priority since borg v2 is already in beta. I created a branch borg-v2-support which smaller PR implementing one of the changes listed above can be based on. I will find some time for such PRs soon but maybe I am not the only one.

[m3nu]

Thanks! Good stuff. I'll find some time too.

[m3nu]

Pushed the first commit to the new v2-branch. This already covers creating and listing a repo. 🎉

For now I'm positive that we can cover the changes with our existing compatibility setup, if it's mostly commands that change and JSON output stays largely the same.

Also adjusted BorgBase.com to better work with v2:

• repo URL is now given in ssh:// format
• you can select Borg v2.0.0b1 as repo version for testing

Also wanted to remind everyone here (@sophie-h @real-yfprojects @witten) that I'll give you a free BorgBase plan if you can use one. Just send your registered email to [email protected].

[real-yfprojects]

Pushed the first commit to the new v2-branch. This already covers creating and listing a repo. 🎉

For now I'm positive that we can cover the changes with our existing compatibility setup, if it's mostly commands that change and JSON output stays largely the same.

Great! The existing compatibility system is already quite powerful. Though I wouldn't recommend using the new branch as a working branch if multiple people commit to it.

Also wanted to remind everyone here that I'll give you a free BorgBase plan if you can use one. Just send your registered email to [email protected].

Thanks. That's very generous.

[m3nu]

Done converting most commands. It wasn't much work at all. Just rename took longer because we had a sloppy setup from the start.

I expect some detailed differences to surface over time. E.g. deduplicated, compressed size seems to be missing from the result of create.

And borg mount isn't fully changed in Borg yet. No way to pass an archive name.

[witten]

And borg mount isn't fully changed in Borg yet. No way to pass an archive name.

My read of the Borg 2 mount sub-command is that if you want to filter down to a particular archive, you just pass in --glob-archives thatarchive. The resulting mounted path isn't exactly the same as when selecting the archive with Borg 1's mount, but it's close enough. (It just has an extra directory with the archive name.) Did you see indication elsewhere that they'd be adding more than --glob-archives?

[m3nu]

Would make sense, since I often want to look at a few versions. So we could allow selecting multiple archives in the UI.

[real-yfprojects]

Do we need a button for mounting the whole repository then? The user could select all archives instead.

[m3nu]

Maybe 2 Mount buttons: One in archive actions, the other in repo actions? Details explained in tooltip.

[real-yfprojects]

Maybe 2 Mount buttons: One in archive actions, the other in repo actions? Details explained in tooltip.

We keep the status quo then.

[real-yfprojects]

• [ ] scp style repository syntax is gone in favor of ssh://. Convert existing URLs in DB?

I would include that in the transfer process. Syntax definition can be found here:

https://github.com/borgbackup/borg/blob/b2eaa2fba46cc12a5bcbebe0a23f3701c893de5b/src/borg/helpers/parseformat.py#L353-L370

[real-yfprojects]

So we could allow selecting multiple archives in the UI.

Currently if one mounts a single archive, the mount point is shown in the GUI for that archive. If the user mounts multiple archives, should that be counted as mounting the full repo or should the mount point show up in the table for each mounted archive?

[m3nu]

Let's keep the current functionality and treat this as new feature later, if we or others see use for it. Could be useful to find some lost files where I don't know the archive exactly. But still mount it faster than when mounting all archives.

[m3nu]

Looks like using ssh://-style paths is the main issue left here? And the conversion feature maybe. If so I'll add the former and then merge this branch, so we can add the rest ad-hoc later.

[witten]

FYI Borg 2.0.0 beta 3 includes breaking --glob-archives changes!

[m3nu]

Yeah, good summary here:

https://borgbackup.readthedocs.io/en/2.0.0b3/changes.html#version-2-0-0b3-2022-10-02

Added a few more things to check above.

[m3nu]

Pushed another batch of adjustments. All features seem to work with the latest beta3 now. 🙌

There is one instance of glob-archives left in the function that finds existing mounts.

And the issue of existing SCP-style URLs. I wonder if we should even touch those, since the repo formats are different too. So would be better to convert them during conversion, if we offer such a feature. New v2 repos will only accept the new URL format anyways.

[real-yfprojects]

And the issue of existing SCP-style URLs. I wonder if we should even touch those, since the repo formats are different too. So would be better to convert them during conversion, if we offer such a feature. New v2 repos will only accept the new URL format anyways.

When using a borg v1 repo with borg v2, vorta should ask whether it should convert it for use with borg v2. If the user dismisses, the repo won't be usable as long as borg v2 is installed.

[real-yfprojects]

Running an operation on a v1 repository with borg v2 will give the rc 2 and the following error message:

> borg rlist
Error: This borg version only accepts version 2 repos for -r/--repo. You can use 'borg transfer' to copy archives from old to new repos.

[m3nu]

That's good to avoid doing something stupid between versions. Can look into adding the transfer command later.

Else, anything important missing to merge the v2 branch? So we can already support both versions.

[real-yfprojects]

Else, anything important missing to merge the v2 branch? So we can already support both versions.

Yes, there is. I have prepared a branch for solving #1409 that is based on this branch since there are some changes needed for both issues. Also I would wait for the final release of borg v2 because there might be another beta that requires changes to this PR.

[m3nu]

That's true. Though the speed of change should slow down with more beta versions.

[real-yfprojects]

I am not sure whether/how we should implement borg transfer since it involves the following steps:

# 0. Have borg 2.0 installed on client AND server, have a b12 repo copy for testing.

# 1. Create a new "related" repository:
# here, the existing borg 1.2 repo used repokey-blake2 (and aes-ctr mode),
# thus we use repokey-blake2-aes-ocb for the new borg 2.0 repo.
# staying with the same chunk id algorithm (blake2) and with the same
# key material (via --other-repo <oldrepo>) will make deduplication work
# between old archives (copied with borg transfer) and future ones.
# the AEAD cipher does not matter (everything must be re-encrypted and
# re-authenticated anyway), you could also choose repokey-blake2-chacha20-poly1305.
# in case your old borg repo did not use blake2, just remove the "-blake2".
borg --repo       ssh://borg2@borgbackup/./tests/b20 rcreate \
       --other-repo ssh://borg2@borgbackup/./tests/b12 -e repokey-blake2-aes-ocb

# 2. Check what and how much it would transfer:
borg --repo       ssh://borg2@borgbackup/./tests/b20 transfer --upgrader=From12To20 \
       --other-repo ssh://borg2@borgbackup/./tests/b12 --dry-run

# 3. Transfer (copy) archives from old repo into new repo (takes time and space!):
borg --repo       ssh://borg2@borgbackup/./tests/b20 transfer --upgrader=From12To20 \
       --other-repo ssh://borg2@borgbackup/./tests/b12

# 4. Check if we have everything (same as 2.):
borg --repo       ssh://borg2@borgbackup/./tests/b20 transfer --upgrader=From12To20 \
       --other-repo ssh://borg2@borgbackup/./tests/b12 --dry-run

If we don't implement the transfer process, vorta should show a descriptive error message (without the notice to file a bug report) that explains how to proceed.

[m3nu]

Yeah, looks like a big hassle and potential to break things. Users who want to convert can do so im the CLI. We always kept complex operations for the CLI. Vorta is meant to run the most frequent commands in an easy way, not 100%.

For BorgBase, we already have versioning, so v1 repos will stay on the v1 branch. New repos will be v2 by default at some point.

Server-side conversions would be tricky too, since some larger repos exceed available space or have file system quotas applied.

[real-yfprojects]

Server-side conversions would be tricky too, since some larger repos exceed available space or have file system quotas applied.

For users of BorgBase it would be great if BorgBase provided a way to update a repo to v2 in place since their quota probably won't allow for a manual transfer using two repos. The operation will still need twice as much disk space as already used but since it is only temporary the server could handle that internally.

[real-yfprojects]

[] scp style repository syntax is gone in favor of ssh://. Convert existing URLs in DB?

Because we won't implement borg transfer, I would convert those urls automatically when borg v2 is installed. AFAIK the ssh:// urls are aquivalent to scp:// urls.