FR: Declarative repo management / adding repo UX improvement ideas

[kodek]

Note: if you have a question or want discussion please post in the discussions area.

Is your feature request related to a problem? Please describe.

I am using VolSync in Kubernetes with a bunch of Restic repos. Backrest has been a great way to inspect snapshots, fix locking problems, etc.

All of the repos are in a single AWS bucket under various subdirectories, so all repo configs are effectively identical. Unfortunately, adding them to Backrest is a bit of a repetitive, error-prone operation. Passing secrets via env substitution definitely, helps, but it still requires editing multiple textboxes. Furthermore, the repo path does not seem to allow env substitution.

Describe the solution you'd like A few potential solutions (apologies for the firehose of ideas!):

• Ideally, it'd be great to manage the repos in a more declarative fashion. A Kubernetes operator using the Backrest API, or some sort of integration with Volsync would be amazing. That said, I realize this isn't the main focus of Backrest. (Perhaps this could be a separate project, though!)
• A second-best option would be a stable, backwards-compatible config schema -- separate from the main config used by Backrest. If I can provide the repos in separate config files, I could generate these declaratively, mount a separate read-only directory, and I would not have to worry about breaking other internal options in config.json.

Or some UX/UI alternatives:

• Perhaps we could duplicate existing repos, or define repo templates, as a way to pre-fill the form.
• Or perhaps the JSON preview could be editable -- then I could paste all the boilerplate in a single textbox. If it's a valid object, it could update the form fields.

Tangentially, if the repo path could support env substitution, I could do something like ${REPO_PATH_PREFIX}/foo/bar. With this, and perhaps some way of inferring RESTIC_PASSWORD, AWS... without adding them explicitly, adding a new repo would only require editing a single textbox.

Thank you!

[garethgeorge]

I've added some tentative support for automatically initializing repos in https://github.com/garethgeorge/backrest/pull/650 which is likely a step in this direction.

Backrest provides a protobuf schema at https://github.com/garethgeorge/backrest/blob/main/proto/v1/config.proto and the configuration does take some pains to preserve backwards compatibility. I'm a bit hesitant to split the config into multiple files as there are references within the configuration structure that must be internally consistent e.g. each plan is linked to a repo, etc. And this complexity grows with the introduction of permissions (oriented around repos) with multihost synchronization which is still on the long term roadmap.

I am happy to think through what generating large configs or merging changes in can look like however.

I do like the suggestion of providing an editable JSON editor in the web UI, I'll give some thought to how this might look / be implemented.

[ericukk]

I have a similar scenario with over 20+ Docker Compose services. Many of these share similar backup configurations. Due to the manual setup process and lazyness, I've only configured backups for the most critical ones.

An editable JSON format would be beneficial, as it would allow me to easily copy configurations from existing services when creating new services.

In the meantime, would you recommend manually editing the config.json file as a temporary solution?

[garethgeorge]

The JSON format is fine to generate programmatically, I'd recommend generating the proto-JSON library for the config.proto in your language of choice and using that to manipulate it.

[sebt3]

I was generating the configuration file for this need. But 1.9 introduced an ed25519 keypair in the config file but since my setup makes the config file read-only. I'm kind of blocked to 1.8 for now. Is there a way to have 2 files ? one for the repo list and one for everything else ?