Exclude BiT files from the backups by default?

[aryoda]

As part of discussions regarding #1003 the idea came on the table if BiT files should be excluded (by default) from backups:

Consider excluding BiT files from the backups

BiT uses rsync and this is not meant to backup all files of the whole system since open files may cause problems. but users want to to do this anyhow so ignoring BiT files in backups would be a good idea.

From what I see in the code

https://github.com/bit-team/backintime/blob/bff7aeb9fd07fb44f82ba5ba4ef2da5f0cc1d56c/common/config.py#L184-L189

the following folders should be excluded too by default (and manually as workaround for older versions of BiT):

• /root/.local/share/backintime
• /<currentuser>/.local/share/backintime

Ignoring specific files kept open by BiT would be even better since ignoring the whole folder means that

• the configs (backup profiles) and
• the user-callback script

are not contained in the backup :astonished:

This would require to add a positive list of all file names (or extensions) that are uses in the config.py (e.g. worker*.* files).

The default config can be changed here in the source code:

https://github.com/bit-team/backintime/blob/bff7aeb9fd07fb44f82ba5ba4ef2da5f0cc1d56c/common/config.py#L132-L135

[Germar]

BiT will backup the configuration in an extra step even if it is not included

https://github.com/bit-team/backintime/blob/110d82aa1cf6717424c39b72366326b5977e3088/common/snapshots.py#L905

[aryoda]

TODO: We should test if the existing config file is overwritten if it is contained in the snapshot and the home folder is restored

[Hitro147]

Hi Developers,

I'm new to open-source contributing and am very excited about contributing to the Back In Time project. However, I've run into a hurdle that I hope you guys can help me with.

I'm currently operating on a Mac, and since Back In Time is designed for Linux, I've been using Docker to create a compatible environment. Also, I am not using the UI version of the project but rather the command line interface through Docker.

Despite following the instructions for taking a snapshot, I continually encounter an error stating that the snapshots folder cannot be found. I've made several attempts to resolve this by creating directories in different locations and adjusting the snapshot path ('profile1.snapshots.path') in the config file. I've experimented with various directory names and paths, but the application still fails to recognize the snapshots folder, and I continue to receive the same error message.

I've attached screenshots illustrating the directory structures I've tried and the error messages encountered.

Any insights into a naming convention, specific directory location, or Docker-related adjustments that I should be aware of would be greatly appreciated.

Thank you for your time and assistance.

[aryoda]

Welcome here @Hitro147

Could we please continue this conversation in the new issue #1622 (which I have just created) to avoid "hijacking" this issue with a completely off-topic discussion.

Thanks a lot!

[Hitro147]

TL;DR: Configuration File Overwrite Behavior in Back In Time

• Issue: Post-snapshot modifications to the .config/backintime/config file are overwritten upon restoring from a snapshot.
• Test: Conducted a test involving a snapshot, modification of the config file, and a subsequent restore operation.
• Implications: Important user configurations made post-snapshot can be lost without warning.
• Recommendations: Implement user warnings, allow selective file restoration, and enhance documentation to address this issue.

---

Machine Configuration and Test Procedure

Machine Configuration

• Operating System: Ubuntu 22.04.3 LTS
• Back In Time Version: 1.4.2-dev
• Docker Version: Docker version 24.0.6, build ed223bc
• FileSystem Layout: Using Docker volumes for /root/snapshots
• Additional Notes: Running Back In Time without GUI; all operations performed via command line

Detailed Steps

Step 1: Initial Backup Creation

• Environment setup and verification of snapshot creation.
• Command: /backintime/common/backintime --config /root/.config/backintime/config --profile-id 1 backup

Step 2: Post-Snapshot Modification

• Modified /root/.config/backintime/config by adding a comment # Modified for Issue #1382 testing and altering a configuration setting (Changed profile1.snapshots.automatic_backup_mode from 0 to 1).
• Documented changes for comparison.

Step 3: Restoration Process

• Initiated restore from the specific snapshot, observing the overwrite of post-snapshot modifications.
• Command: /backintime/common/backintime --config /root/.config/backintime/config --profile-id 1 restore

Observations

Upon restoration, the .config/backintime/config file reverted to its snapshot state, indicating that modifications made after the snapshot were overwritten.

Implications

This behavior poses a risk of unintentional data loss, especially for users who might not expect configuration changes to be overwritten during restoration.

Recommendations

1. User Warnings: Introduce warnings or prompts during the restoration process to alert users to potential overwrites.
2. Selective Restoration: Develop options to selectively restore or exclude files, providing users more control.
3. Documentation Enhancements: Clearly describe overwrite behavior in documentation, offering guidelines to protect important changes.

Conclusion

The test confirms that Back In Time's current restoration process overwrites files with their snapshot versions, specifically the configuration file. Addressing this through improved warnings, selective restoration options, and better documentation can enhance the tool's usability and reliability.

[buhtz]

Dear @Hitro147 , your last comment reads like a new issue. Please open a new issue instead of posting it here. It is of topic to this issue.

[Hitro147]

Hey @buhtz,

My last comment contains observations and testing results for the TODO mentioned in @aryoda's last comment.

Let me know if I should open a new issue referencing to this issue as parent :)

[emtiu]

This actually fits very well here. We should check whether we consider this a bug.

[buhtz]

That was what I had in mind. It is related to the current Issue but an Issue of its own.

Hitro147: Do I understand it correct, that the config file was not an entry in the regular "Include" list of that backup? But it was included by default because of the bahvior described by Germar? I am asking because I am not able to reproduce this behavior here.

Germar do you remember why this behavior (restoring config files) was introduced?

[Hitro147]

The config file was indeed included in the snapshot by default behavior. It was not explicitly listed in the regular "Include" list for the backup profile. The inclusion happened as part of the default behavior described by Germar, where Back In Time includes the entire home directory (/root in this case) by default, which contains the .config/backintime/ directory and its contents, including the config file.

Perhaps ensuring that the entire home directory is included by default in your test setup might help replicate the behavior?

[emtiu]

The config file was indeed included in the snapshot by default behavior. It was not explicitly listed in the regular "Include" list for the backup profile. The inclusion happened as part of the default behavior described by Germar, where Back In Time includes the entire home directory (/root in this case) by default, which contains the .config/backintime/ directory and its contents, including the config file.

That's not what Germar was talking about, though. He is referring to a functionality that includes the config even if $HOME/.config/backintime is not part of the included paths. If you inspect your snapshot directories, you will find this file like host/user/profile/20230131-230001-815/config, written by this function that Germar is referring to:

https://github.com/bit-team/backintime/blob/110d82aa1cf6717424c39b72366326b5977e3088/common/snapshots.py#L905

As I see it, we have established two useful facts:

1. BackInTime's config is stored with any snapshot, as in host/user/profile/20230131-230001-815/config. This is useful and poses no problem.
2. If the home directory is part of a snapshot, and it is restored with default settings, this will overwrite $HOME/.config/backintime, potentially causing trouble (as tested by @Hitro147). This is something we might want to address.

(Side note: Might the config also live somewhere other than $HOME/.config/backintime? If yes, we need to account for that.)

How might we proceed?

1. Warning the user of the consequence of overwriting the config location: probably a bit tricky to code, also the user would be presented with this at a stressful time (=when restoring $HOME). Also, should the user then be presented with any choices—and if yes, which? Not a great solution.
2. Hardcoding an exclude for the config location: not cool, potentially unexpected behavior.
3. Including the config location with the default excludes, manageable by the user: minimally invasive, easy to understand, probably the most elegant solution.

I prefer 3. Which is what @aryoda has been proposing right from the start ;)

[Hitro147]

Oops! My bad. I totally got the wrong idea. Thanks for clearing that up and helping me see it right 😄

[buhtz]

Thanks emtiu for your summarize. My apologize I was on the total wrong track here.

1. Warning the user of the consequence of overwriting the config location: probably a bit tricky to code, also the user would be presented with this at a stressful time (=when restoring $HOME). Also, should the user then be presented with any choices—and if yes, which? Not a great solution.
2. Hardcoding an exclude for the config location: not cool, potentially unexpected behavior.
3. Including the config location with the default excludes, manageable by the user: minimally invasive, easy to understand, probably the most elegant solution.

I would like to add a number 4 proposal similar to number 3.

Don't add the config location to the default exclude. But when the users add new Include entries to the list, BIT should detect if its own config file is involved. In that case the user should get a question-dialog-box with an explanation (BIT config files are included), a warning (overwritten on restore) and a decision (add config files to Exclude list? Yes/No). The dialog should show up directly when the "Add Include" was done. To my understanding this should be easy to implement.

[emtiu]

Don't add the config location to the default exclude. But when the users add new Include entries to the list, BIT should detect if its own config file is involved. In that case the user should get a question-dialog-box with an explanation (BIT config files are included), a warning (overwritten on restore) and a decision (add config files to Exclude list? Yes/No). The dialog should show up directly when the "Add Include" was done.

I like that.

The dialog box could basically say: „Attention: You're including BiT's own config in the snapshots. Please be aware: Restoring from these snapshots to your running system later might overwrite BiT's configuration with an older state, leading to unexpected behavior. If you know what you're doing, you may continue. Otherwise, BiT can add the location of its own configuration from the snapshots for you. You can always adjust this behavior through the Include/Exclude lists.“

Two buttons: „Snapshots should include BiT config“ and „Exclude BiT config from snapshots“.

The wording is not great yet: we must be very carfeul to make it understandable and not too complicated. I'll be happy to do later—writing understandable words about complicated things is my profession ;)