Exclude patterns GUI improvements

[m3nu]

Just noticed that Nextcloud (desktop client) comes with a nice list of default exclusions. Maybe something for Vorta to assimilate.

I agree that one simple kind of patterns would be enough.

Originally posted by @m3nu in https://github.com/borgbase/vorta/discussions/838#discussioncomment-369487

[erAck]

BackInTime also has a sensible set of default excludes:

• .gvfs
• .cache/*
• .thumbnails*
• [Tt]rash*
• .backup
• *~
• .dropbox*
• /proc/*
• /sys/*
• /dev/*
• /run/*
• /etc/mtab
• /var/cache/apt/archives/*.deb
• lost+found/*
• /tmp/*
• /var/tmp/*
• /var/backups/*
• .Private

[m3nu]

We discussed adding defaults before and the feedback wasn't good due to the large variety of OS and use cases.

So if we add defaults, they will be OS-specific and disabled by default.

[m3nu]

Suggested by @Wikinaut in #970

I cannot find that borg option --exclude-caches to be used by vorta. Was that intended?

See https://borgbackup.readthedocs.io/en/stable/usage/create.html?highlight=exclude%20caches

I suggest to add this as a vorta option which is active by default.

[m3nu]

Getting close to starting this feature. Was thinking about this a bit more and would want those features:

• Maintain templates of exclusions and enable them for selected profiles. Like "Dev Cache Files" or "App Cache Files".
• Currently I use 2 groups of exclusions: specific paths (e.g. ~/Library/BlahBlah/Cache and patterns that should be matched everywhere (e.g. */.git)
• Maybe we also want "Exclude folders with this file"
• Add description to exclusion
• Import and export exclusions as CSV? (pattern, desc)

Anything to add? @Hofer-Julian @bastiencyr ?

[Hofer-Julian]

Sounds good. One idea:

• Add description to exclusion
• Import and export exclusions as CSV? (pattern, desc)

How about allowing comments with #? In that case you don't need an extra field for descriptions and the export format would be a list of patterns and comments instead of a csv.

[m3nu]

I like the idea of making the import/export file a valid patterns file.

In the UI, I'd prefer to have pattern and commend separate. Else we literally have a free text box again.

[bastiencyr]

Souds good too.

Maintain templates of exclusions and enable them for selected profiles. Like "Dev Cache Files" or "App Cache Files".

For naming, maybe some more meaningful names for users like "User Temporary Files", "Dev Irrelevant Files". Its more user orientated and may not know what a cache is.

[m3nu]

macOS via arkTanlis on Reddit.

(Some wildcards used here are not appropriate for the default fnmatch patterns we use.)

~/.Trash
**.DS_Store
~/Library/Caches
~/Library/Logs
**.cache
~/.npm
~/Library/Containers
**.git
**node_modules

[m3nu]

Adding my own excludes:

*/.DS_Store
*/._sync_*
*.pyc
*.mp4
*.mkv
*/.~*
*-journal
*/.owncloudsync.log
*/.cache
*/__pycache__
~/Pictures/20*
~/Pictures/Phone

[m3nu]

Some more inspiration from Back in Time. They probably keep a list for Linux.

[real-yfprojects]

I sketched a first GUI mock-up. What do you think?

[m3nu]

I think it's great. 🙌 Not much to add.

So the data model is like description, pattern, OS, on/off? Some of those are built-in and some are supplied by the user? We should put the built-in patterns in a reusable format, since they may be relevant for other projects, if we end up collecting enough.

[Hofer-Julian]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

[real-yfprojects]

So the data model is like description, pattern, OS, on/off? Some of those are built-in and some are supplied by the user? We should put the built-in patterns in a reusable format, since they may be relevant for other projects, if we end up collecting enough.

There patterns that can be grouped together with a name. They can be toggled through the GUI or by commenting them out in raw text. The predefined pattern groups that have tags (if they are app, os specific,...) and possibly a description. There is a search bar for searching through the patterns/groups by group name and tags as well as pattern contents.

We can integrate online sources for patterns like gitignore.io or github/gitignore for excluding different development files.

[real-yfprojects]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

I had a look at Pika Backup. It seems like they only allow excluding existing files/folders that can be added to the exclusion list through a file chooser.

[m3nu]

That's not very flexible. E.g. to exclude a type of files.

[Hofer-Julian]

https://gitlab.gnome.org/World/pika-backup/-/merge_requests/69 might be interesting for this effort too

I had a look at Pika Backup. It seems like they only allow excluding existing files/folders that can be added to the exclusion list through a file chooser.

That specific feature is not released yet. But that screenshot should give you an idea: https://gitlab.gnome.org/World/pika-backup/-/issues/94#note_1504221

[m3nu]

Yeah, having multiple patterns in a group makes sense too.

So the data struct would be like: group description, [list of patterns], OS (as per Python sys.platform), on/off

[real-yfprojects]

Yeah, having multiple patterns in a group makes sense too.

So the data struct would be like: group description, [list of patterns], OS (as per Python sys.platform), on/off

Yes, that would be a good start. However the user should be able to toggle single patterns, edit the predefined patterns and add own patterns. This could be realised in two database tables: PatternGroup and Pattern. However my idea was getting the data from a raw text file/entry the user can edit.

That specific feature is not released yet. But that screenshot should give you an idea: https://gitlab.gnome.org/World/pika-backup/-/issues/94#note_1504221

Thanks. I like that the mockup shows the size of the excluded files.

[fooness]

Hello everyone. Could you please point me to the current list of different exclude defaults/suggestions? Maybe I missed the link here, or did search for the wrong keywords. I thought I could look the exclusions up here in the repo, without installing Vorta (as I’m basically using borg without GUI for my macOS backups).

---

Here’s what I came up with for now; it’s a mixture of Arq’s exclusion list as well as the suggestions from the previously linked reddit thread and from the posts in this very GitHub issue … comments and feedback very much appreciated; this is macOS-specific.

**/.*.swp
**/.DS_Store
**/.VolumeIcon.icns

# not quite sure what the following two exclusions are for
**/.file
**/.vol

**/.DocumentRevisions-V100/
**/.MobileBackups/
**/.MobileBackups.trash/
**/.Spotlight-V100/
**/.TemporaryItems/
**/.Trash/
**/.Trashes/

**/.cache/
**/.fseventsd/
**/.npm/

**/Library/**/Cache/
**/Library/**/Caches/
**/Library/**/Containers/
**/Library/**/CrashReporter/
**/Library/**/DerivedData/
**/Library/**/Developer/
**/Library/**/Metadata/
**/Library/**/MobileSync/
**/Library/**/Logs/

**/iTunes/

**/build/
**/dist/
**/node_modules/
**/tmp/
**/vendor/

/System/
/Users/Shared/
/Volumes/
/bin/
/cores/
/dev/
/etc/
/home/
/macOS Install Data/
#/nix/
#/opt/
/private/
/sbin/
/tmp/
#/usr/
/var/

# comment: /nix/ + /opt/ + /usr/ maybe should not be excluded due to e.g. homebrew …

[m3nu]

Thanks for sharing! We don't ship a default list yet. Still work in progress. Based on the current plan, we may do different groups that can be toggled. Since exclusions are really OS-dependent and personal.

[fooness]

Based on the current plan, we may do different groups that can be toggled. Since exclusions are really OS-dependent and personal.

Yes, the definitely are; been looking for the macOS-specific list.

We don't ship a default list yet. Still work in progress.

That’s the part I obviously have missed reading through this issue; will keep an eye on future updates.

PS: I’ll be doing some more research on this matter, and in case of any new insights, I’ll be editing/updating my list above.

[diivi]

I made a design for the exclude GUI, it's available here and is awaiting community feedback. I'll make changes to it based on the response received. Please let me know what can be improved! I tried to keep it visually pleasing and beginner friendly, while still giving the user some powerful options. Attaching a quick screenshot here too:

• Presets can be selected in the Add Dialog to make this a little less confusing for beginners (Simple by default, powerful when needed.)
• Multiple files and folders can be added at once instead of clicking the add button again and again.
• Comments can be added to the file/folder being added, these will show up in the Raw tab and will be ignored in the Patterns tab.
• If user selects a preset, the patterns field becomes disabled.
• Raw tab will show all the rules, kinda like the current pattern exclusion field, but will include comments and have functionality to import/export a set of rules as txt, as @Hofer-Julian mentioned.
• Inspired by mockups made by @real-yfprojects, but I removed the separate patterns and presets tab, as I thought this might be simpler for most users (I couldn't differentiate between the two tabs and too much functionality at once seemed overwhelming)

[erAck]

Looks good in general.

If user selects a preset, the patterns field becomes disabled.

I'd rather display the preset's content then and have it editable. For that, the Pattern text edit should be larger and popup resizable.

What should happen if a(nother) preset is selected while patterns (in the main dialog) already exist? Replace, after warning? Merge (add without duplicates)? Needs further buttons? Or shouldn't rather a preset choice only be presented if there are no excludes yet?

What does the bottom field with /tmp/my_file tell? What file is that? The file the excludes are saved to? And what does the circled checkmark indicate? File name is valid? Or would it be a single pattern's edit field? Unclear.

The Add pattern to exclude popup shouldn't have a Save button, because that's not what it would do (or would it? confusing), it would add the pattern(s), so maybe Add instead? The only real Save is on the main dialog. (I hope).

Nitpicking: File names starting with # are valid file names, even # with a blank following, even more they might be wanted to be excluded. How to resolve the conflict?

[diivi]

Thanks for the detailed review! Very helpful.

I'd rather display the preset's content then and have it editable. For that, the Pattern text edit should be larger and popup resizable.

Yes, I can do that, afterall there's no harm in making them editable (in the field) and it adds functionality.

What should happen if a(nother) preset is selected while patterns (in the main dialog) already exist?

I was thinking if a pattern present in a preset or a pattern in general already exists in the patterns tab, it would not be replaced. So yes, for a mixed case, merge. I don't think a warning in the UI would be necessary when merging though.

What does the bottom field with /tmp/my_file tell?

Ah, agreed that it's not clear what this does, but do you think replacing the placeholder text with "Enter a path pattern to be matched" helps (I just missed changing the placeholder text). Explanation - /tmp/* was excluded and /tmp/my_file was entered, hence the check mark. This is taken from the mockups @real-yfprojects shared earlier.

The Add pattern to exclude popup shouldn't have a Save button, because that's not what it would do

True. Will change this, thanks.

File names starting with # are valid file names, even # with a blank following, even more they might be wanted to be excluded. How to resolve the conflict?

Good point, for now I can just think of removing the option to add comments in that field and keep that only for the Raw tab. Open to other ideas though.

[real-yfprojects]

I added some comments on figma. Some design choices I would like to discuss:

What about having a preset list instead of a checkbox? This will make it much more comfortable to look through and select a preset.

In the current mockup one isn't able to determine which rules an added preset defines without opening the raw tab.

I somewhere read about dropping the raw tab. I like it since it allows power users to add/edit patterns fast as well as copy and pasting them. I would especially appreciate the option to add comments so that I can remember what a pattern does when I come back to a profile after some time. One could even have a syntax for defining custom groups (e.g. #%% group_name) so that the graphical list doesn't get to long and cluttered.

File names starting with # are valid file names, even # with a blank following, even more they might be wanted to be excluded. How to resolve the conflict?

See borg docs:

The --exclude-from option permits loading exclusion patterns from a text file with one pattern per line. Lines empty or starting with the number sign (‘#’) after removing whitespace on both ends are ignored. The optional style selector prefix is also supported for patterns loaded from a file. Due to whitespace removal, paths with whitespace at the beginning or end can only be excluded using regular expressions.

So whitespace trimming should also be done by vorta automatically whenever the user adds/edits patterns.

What does the bottom field with /tmp/my_file tell?

Ah, agreed that it's not clear what this does, but do you think replacing the placeholder text with "Enter a path pattern to be matched" helps (I just missed changing the placeholder text). Explanation - /tmp/* was excluded and /tmp/my_file was entered, hence the check mark. This is taken from the mockups @real-yfprojects shared earlier.

This was the solution I came up with for checking whether one configured excludes correctly. I suggest the tooltip shows Check a path for exclusion. On entering vorta will try to match the path with all patterns. If a matching pattern is found a check mark is displayed else a cross. The check mark could be implemented using QToolButton, it would be clickable and highlight the matching pattern in the pattern list.

Yes, I can do that, afterall there's no harm in making them editable (in the field) and it adds functionality.

The preset should change to None automatically when editing a preset.

[diivi]

I added some comments on figma.

Checked and made some edits.

What about having a preset list instead of a checkbox? This will make it much more comfortable to look through and select a preset.

Can you share a sample UI that I can refer to? Is it something like this? (changing the active list item will repopulate the text field contents, and editing the text field will make it switch back to "None") I agree this might be easier than opening the combobox again and again to view different presets. Will make the add dialog a bit bigger though, but I guess that would be ok.

In the current mockup one isn't able to determine which rules an added preset defines without opening the raw tab.

Is that information necessary for the user? I feel like the presets should just be there to assist the user in getting pre-defined patterns. If they really want to see why a pattern is there they can just look in the Raw tab. I wanted to keep the Patterns tab less cluttered and the simplest thing a user would want.

[real-yfprojects]

Is it something like this?

Yes, exactly.

I feel like the presets should just be there to assist the user in getting pre-defined patterns. If they really want to see why a pattern is there they can just look in the Raw tab. I wanted to keep the Patterns tab less cluttered and the simplest thing a user would want.

When adding a preset do want to show only the preset name in the pattern list or an entry for each pattern in the preset?

[diivi]

When adding a preset do want to show only the preset name in the pattern list or an entry for each pattern in the preset?

I was thinking an entry for each pattern in the preset, to let me disable singular patterns too. But if you would like the preset to be there as a group, maybe we can list all the presets first, and then singular patterns?

[real-yfprojects]

I was thinking an entry for each pattern in the preset, to let me disable singular patterns too.

Ah, that wasn't clear since your mockup didn't contain patterns only.

But if you would like the preset to be there as a group, maybe we can list all the presets first, and then singular patterns?

I think there will be a lot of patterns pretty fast. There should be a way to organize them.