snapshot-policy USN plus attribute exclude may exclude all selected files

[ts678]

• [x] I have searched open and closed issues for duplicates.
• [x] I have searched the forum for related topics.

---

Environment info

• Duplicati version: 2.0.7.1 Beta
• Operating system: Windows 10
• Backend: Local folder

Description

Using Usn policy can cause loss of files discovered a bad interaction between these settings that seemingly decides there are no files. Testing with buttons on GUI Source Exclude, it looks like System files and Hidden files can do it, but Temporary files is safe.

Steps to reproduce

1. Make a test folder somewhere, e.g. C:\tmp\usntest
2. Put a file in it, e.g. A.txt containing an A.
3. Select the file in GUI Source screen tree.
4. In its Exclude section, check System files.
5. In Advanced options, set snapshot-policy to something that will use USN. Of course, you must run as an elevated administrator.
6. Backup.

• Actual result: No files. Home screen Source size is 0 bytes, Restore tree area empty.
• Expected result: File backed up.

If I select the containing folder, e.g. C:\tmp\usntest here, the file gets backup.

Screenshots

Debug log

Forum has a few. The most worrisome part of a verbose log is when you see:

2024-05-04 19:46:47 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-ExcludingPathFromAttributes]: Excluding path due to attribute filter: C:\

[duplicatibot]

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/using-usn-policy-can-cause-loss-of-files/17878/23

[Jojo-1000]

I can't reproduce this. In the forum post you had this sequence

2024-05-04 19:46:47 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\Users\Maintenance\aranym\.matplotlib\
2024-05-04 19:46:47 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\Users\Maintenance\aranym\
2024-05-04 19:46:47 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\Users\Maintenance\
2024-05-04 19:46:47 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingPath]: Including path as no filters matched: C:\Users\
2024-05-04 19:46:47 -04 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-ExcludingPathFromAttributes]: Excluding path due to attribute filter: C:\

I only get a single line with

2024-05-14 18:58:31 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\duplicati\test\changelog.txt

When I add the drive root as the source (with a filter to exclude subdirectories), I get the warning

2024-05-14 19:01:57 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\
System.IO.IOException: Incorrect Parameter. : '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy19\'
   at System.IO.File.GetAttributes(String path)
   at Duplicati.Library.Common.IO.SystemIOWindows.GetFileAttributes(String path) in Duplicati\Library\Common\IO\SystemIOWindows.cs:line 328
   at Duplicati.Library.Snapshots.WindowsSnapshot.GetAttributes(String localPath) in Duplicati\Library\Snapshots\WindowsSnapshot.cs:line 223
   at Duplicati.Library.Utility.Utility.EnumerateFileSystemEntries(String rootpath, EnumerationFilterDelegate callback, FileSystemInteraction folderList, FileSystemInteraction fileList, ExtractFileAttributes attributeReader, ReportAccessError errorCallback)+MoveNext() in Duplicati\Library\Utility\Utility.cs:line 390
2024-05-14 19:01:57 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\
2024-05-14 19:01:57 +02 - [Warning-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-FileAccessError]: Error reported while accessing file: C:\
System.IO.IOException: Incorrect Parameter. : '\\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy19\'
   at System.IO.File.GetAttributes(String path)
   at Duplicati.Library.Common.IO.SystemIOWindows.GetFileAttributes(String path) in Duplicati\Library\Common\IO\SystemIOWindows.cs:line 328
   at Duplicati.Library.Snapshots.WindowsSnapshot.GetAttributes(String localPath) in Duplicati\Library\Snapshots\WindowsSnapshot.cs:line 223
   at Duplicati.Library.Utility.Utility.EnumerateFileSystemEntries(String rootpath, EnumerationFilterDelegate callback, FileSystemInteraction folderList, FileSystemInteraction fileList, ExtractFileAttributes attributeReader, ReportAccessError errorCallback)+MoveNext() in Duplicati\Library\Utility\Utility.cs:line 390
2024-05-14 19:01:57 +02 - [Verbose-Duplicati.Library.Main.Operation.Backup.FileEnumerationProcess-IncludingSourcePath]: Including source path: C:\

The resulting backup is empty, so that might be the same reason.

[kenkendk]

@Jojo-1000 Not directly related to the OP question, but would it make sense to automatically exclude \\?\GLOBALROOT\Device\ ?

The data there is unlikely to be restorable, even if there is permissions to read it.

[Jojo-1000]

@Jojo-1000 Not directly related to the OP question, but would it make sense to automatically exclude \\?\GLOBALROOT\Device\ ?

The data there is unlikely to be restorable, even if there is permissions to read it.

I don't think that is necessary. I doubt the normal user will ever enter such a path, so the only time it comes up is with snapshots where it is needed.

[duplicatibot]

This issue has been mentioned on Duplicati. There might be relevant details there:

https://forum.duplicati.com/t/vss-canary-2-0-9-x/18578/4