No incremental backups of VeraCrypt file containers

[kees-z]

I have:

• [x] searched open and closed issues for duplicates

---

Version info

Duplicati Version: 2.0.1.45 Experimental Operating System: Windows 10 Pro Backend: WebDAV

Bug description

I have a VeraCrypt file container of 10 GB included in a backup task. The first time Duplicati runs this task, the complete file container is backed up without problems. However, when the container is mounted and changes are made to the contents of the container file, Duplicati doesn't detect the change and skips this file. Reason for this seems to be that the file modification timestamp is not changed by default and the file size stays exactly the same. So Duplicati should be able to detect changes another way, for example:

• Some other mechanism could be introduced to find out if a file has been modified.
• A file list could be added in a backup task (something like the file extensions list for skipping compression) that always makes a backup of that file. Deduplication should prevent writing anything to the backend if the file container has not been changed.

Steps to reproduce

• Create a VeraCrypt container file and make a backup with Duplicati
• Mount the container file and copy some files to the container.
• Re-run the backup
• Nothing is written, even though the source file has been modified.

[maciekb]

Hi,

I have not used VeraCrypt, but with TrueCrypt was a similar problem. Backup software not detected changes in volume because of setting "Preserve Modification Timestamp". Often backup software detecting changed files first by changed modification time. If modification time was not changed, a file is skipped. Maybe this is the same issue?

Best regards, Maciek

[kenkendk]

The option in Duplicati would be --disable-filetime-check.

[kees-z]

Yes, this seems to be the same issue. Problem is that VeraCrypt (and TrueCrypt) has "Preserve modifications timestamp of file containers" selected by default. If the end user is not aware of this behaviour (Duplicati not detecting that an encrypted file container has been modified), he could be unpleasantly surprised and find out that tons of data have never been backed up, when he needs it and tries to restore the file container.

For me it is not a big deal, I deselected "Preserve modifications timestamp of file containers" in VeraCrypt and Duplicati seems to work well with this setting.

[kees-z]

@kenkendk You beat me by 19 seconds! I'll try that setting soon, but this is probably risky behaviour of a backup solution. The end user probably expects that the software detects every modification of the selected source files. If the --disable-filetime-check indeed works in this situation, this should be the default setting and --enable-filetime-check could be specified to speed up source file processing.

[kees-z]

The --disable-filetime-check indeed detects changes to the file container, even when the timestamp and filesize do not change. Backdraw is that the backup task seems to need a bit more time to complete. However, IMO reliability should have a higher priority than speed, that's why I think this should be the default behaviour which could optionally be toggled with --enable-filetime-check.

[kenkendk]

When you use --disable-filetime-check it will scan every file completely. If you have a large backup this will be a very significant overhead.

The filesystem keeps track of modified files with the timestamps and VeraCrypt (and others) deliberately disable this. Normally the changes would be detected as the file size has changed, but in this case the file stays the same size as well.

I would be more inclined to add a new option that selectively disables the filetime check for some expressions, like the filters. You could then say --disable-filetime-check-for=*.vc.

[gorbiWTF]

I don't think it's a good idea to change the default behaviour for this very specific case. If someone uses Veracrypt they should be aware of how it works and what it does (e.g. the container doesn't change in size).

And as kenkendk said it, Veracrypt "causes" the problem and it also already has the solution for it so I don't think there's a reason to change anything for Duplicati - adding a new optional filter sounds fine though.

Side note: Backing up the entire 10 GB container every time sounds awfully resource-wasting to me.

[kees-z]

This is indeed a very specific issue. It is questionable if Duplicati should have a general performance drop, just to tackle this issue. I don't think so. On the other hand, anyone who uses TrueCrypt/VeraCrypt or similar software in combination with Duplicati, should be aware of this default behaviour. An additional question in the Add-backup wizard should help to let the end user make the correct choice, something like: Option 1: Check Timestamp and Filesize (Fast, works with most backups) Option 2: Check File signature (Slower, but modifications will be detected in files that keep the same file size and timestamp). This also ensures that the end user thinks about which choice is the best option for his environment and - in case of a restore - the end user was expecting that all his files were backed-up, but finds out that a lot of imported data was skipped.

This additional question can be combined with a --disable-filetime-check-for command line option. (BTW: Backing up a 10GB file every time should not be very resource-wasting, deduplication will prevent uploading anything if the file is not modified since the last backup. If the file is unchanged, it will be not more than a check if anything is changed).

[awerpoiu]

Hi,

I am such an Enduser :-) Everyone says use Veracrypt and everyone says use Duplicati. But it's 2020 and I don't find any information beside of this thread, if duplicati meanwhile works without any workflow with truecyrpt/veracrypt (inspecially not in german). Can I use Veracrapt for a folder with photos and duplicati save's just the new ones in the backup? Do i have to expect any trouble with the transfering from the backup? It sound complicated, because it's double-encrypted.

[ts678]

Can I use Veracrapt for a folder with photos and duplicati save's just the new ones in the backup?

I don't use VeraCrypt, but the original issue was about backing up the container. You're asking about the unencrypted files inside the mounted container, correct?

If they work fine with other applications, I'd think Duplicati can back up as usual. Duplicati would backup any new photos, and any changes to preexisting photos.

Do i have to expect any trouble with the transfering from the backup? It sound complicated, because it's double-encrypted.

If you backup photos that VeraCrypt has decrypted and presented at its mount, there would only be Duplicati's encryption done before saving in its backup files.

A restore from Duplicati would unencrypt, and VeraCrypt might reencrypt them. Encryption behavior depends on where you're transferring the files to, of course.

[swimJim]

I apologize if this is not the right place for this, but I'm having a similar problem to the original poster in that I have a Veracrypt file container that I want backed up. I deselected "Preserve modifications timestamp of file containers" but the date of the encrypted volume did not seem to update after I made changes. So I asked ChatGPT to create a script for me that would update the date of the file using the NIR soft command line tool. That part worked. So I ran my Duplicati backup again and tried to restore the file, but I got this warning message:

{
  "RestoredFiles": 0,
  "SizeOfRestoredFiles": 0,
  "RestoredFolders": 0,
  "RestoredSymlinks": 0,
  "PatchedFiles": 0,
  "DeletedFiles": 0,
  "DeletedFolders": 0,
  "DeletedSymlinks": 0,
  "MainOperation": "Restore",
  "RecreateDatabaseResults": null,
  "ParsedResult": "Warning",
  "Version": "2.0.7.1 (2.0.7.1_beta_2023-05-25)",
  "EndTime": "2024-03-04T04:02:45.4240155Z",
  "BeginTime": "2024-03-04T04:02:44.641607Z",
  "Duration": "00:00:00.7824085",
  "MessagesActualLength": 7,
  "WarningsActualLength": 1,
  "ErrorsActualLength": 0,
  "Messages": [
    "2024-03-03 20:02:44 -08 - [Information-Duplicati.Library.Main.Controller-StartingOperation]: The operation Restore has started",
    "2024-03-03 20:02:44 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Started:  ()",
    "2024-03-03 20:02:44 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: List - Completed:  (6.29 KB)",
    "2024-03-03 20:02:45 -08 - [Information-Duplicati.Library.Main.Database.LocalRestoreDatabase-SearchingBackup]: Searching backup 0 (3/4/2024 3:57:44 AM) ...",
    "2024-03-03 20:02:45 -08 - [Information-Duplicati.Library.Main.Operation.RestoreHandler-RemoteFileCount]: 1 remote files are required to restore",
    "2024-03-03 20:02:45 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Started: duplicati-bc0548b07312543dbb9831cd084127b02.dblock.zip (6.84 MB)",
    "2024-03-03 20:02:45 -08 - [Information-Duplicati.Library.Main.BasicResults-BackendEvent]: Backend event: Get - Completed: duplicati-bc0548b07312543dbb9831cd084127b02.dblock.zip (6.84 MB)"
  ],
  "Warnings": [
    "2024-03-03 20:02:45 -08 - [Warning-Duplicati.Library.Main.Operation.RestoreHandler-NoFilesRestored]: Restore completed without errors but no files were restored"
  ],
  "Errors": [],
  "BackendStatistics": {
    "RemoteCalls": 2,
    "BytesUploaded": 0,
    "BytesDownloaded": 7176810,
    "FilesUploaded": 0,
    "FilesDownloaded": 1,
    "FilesDeleted": 0,
    "FoldersCreated": 0,
    "RetryAttempts": 0,
    "UnknownFileSize": 0,
    "UnknownFileCount": 0,
    "KnownFileCount": 6436,
    "KnownFileSize": 164670861315,
    "LastBackupDate": "2024-03-03T19:57:44-08:00",
    "BackupListCount": 20,
    "TotalQuotaSpace": 4000750497792,
    "FreeQuotaSpace": 2736465117184,
    "AssignedQuotaSpace": -1,
    "ReportedQuotaError": false,
    "ReportedQuotaWarning": false,
    "MainOperation": "Restore",
    "ParsedResult": "Success",
    "Version": "2.0.7.1 (2.0.7.1_beta_2023-05-25)",
    "EndTime": "0001-01-01T00:00:00",
    "BeginTime": "2024-03-04T04:02:44.641607Z",
    "Duration": "00:00:00",
    "MessagesActualLength": 0,
    "WarningsActualLength": 0,
    "ErrorsActualLength": 0,
    "Messages": null,
    "Warnings": null,
    "Errors": null
  }
}

Apparently, the encrypted file with the updated date cannot be restored. The command I used to update the date of the encrypted volume was: Run, nircmd setfiletime "[insert file path here]" now now

[kenkendk]

It looks like the container was somehow not included in the restore. This can be because it has not been backed up or because the filter/selection logic somehow excludes it. Can you see the container file in the listing?

I think VeraCrypt and others are not updating the timestamps on the container, and there is no really good way to fix this.

I would suggest that you instead have a backup that includes only the container(s), and then use the option --disable-filetime-check. The flag will make Duplicati ignore the timestamp and do a full scan of all files in the source.

[swimJim]

Yes, the file container is in the list when I go to restore.Also, it appears that VeraCrypt can update the date according to Windows. If you deselect that option, I was just being a goofball. Still though, I am getting the error message about the file not being restored when I tried to restore it.

[swimJim]

I will try the disable file time check flag. I'm surprised in the years since the original poster that the feature where you can just disable the file time check for individual files was not implemented, but I love this backup software very much.

[kenkendk]

I guess it is a fairly trivial task to implement pr-file rules, but it quickly becomes hard to manage details on each file, and tracking down configuration issues.

For the issue, it looks like you are restoring only the directory, not the directory contents?

[swimJim]

My use case is that I have a Duplicati backup that regularly preserves my entire documents folder and the encrypted volume is one file. I was just trying to make sure that the encrypted volume is being backed up, whenever I add new contents to the encrypted volume. So I was only trying to restore that one individual encrypted volume, which is inside my documents folder in Windows 10.

[ts678]

So I was only trying to restore that one individual encrypted volume

Where were you trying to restore it to? If file is already perfect, there is nothing to do, and the confusing warning is presented:

Restore completed without errors but no files were restored

happens on second restore of the same file to a "Pick location". First restore is fine. Second says all is well, without a file restore.

[swimJim]

Okay, I think it worked now, as long as I made sure the original file wasn't in the restore location. I was just trying to make sure that the changes to the encrypted volume are being noticed and backed up appropriately by Duplicati. It seems to be the case, so that is good. But thank you all for your help.

[kenkendk]

I think this means we should change the wording of the warning, to indicate that this situation could be the cause.

[Poikilos]

• A file list could be added in a backup task (something like the file extensions list for skipping compression) that always makes a backup of that file. Deduplication should prevent writing anything to the backend if the file container has not been changed.

• I agree with the file extension filter. People generally will not notice the problem until data is lost. I was using a custom backup program I wrote, which also had this issue, and due to the issue they spend somewhere between $2000-$4000 for data recovery. I did more custom code just to deal with incremental backups of the tc file, but after a disaster is not really a good time to address the issue. Now I switched them to Duplicati, and still have the issue, so they still must run a separate program for the one file.
• Having Duplicati check content of every file would be a big performance hit. A checksum is slow. However:]
  • a byte comparison can be relatively fast
  • even faster if you do 64-bit integer comparison whenever at least 8 bytes are readable such as example code linked at #5523
    • and if you don't have to check the whole file when a difference is found (about 12s on i5 per gigabyte if same, but only about 1 second if differs, probably due to encryption rotating due to changes in the FAT or other header info in the volume).

Please make hc (and tc for TrueCrypt) in such a list and have the list enabled by default.

• ensure timestamping of the file inside the backup is done in some reasonable way--current date when delta was found maybe (but in that case, make sure other code doesn't assume file on backup is newer than local if backup timestamp is newer).

There is a workaround involving a separate backup job within Duplicati (with --disable-filetime-check) for only this one file, but again as noted at #5523, but the issue is very costly as I described (and the issue is also in Microsoft Windows File History formerly called Shadow Copy), so please revisit this old issue and consider a new default for these file types. It is a ticking timebomb.