restic tries to backup its own temp files

[dirk777]

Output of restic version

restic 0.7.2 compiled with go1.8 on windows/amd64

How did you run restic exactly?

restic -r F:\Backup backup C:\Users\USER

What backend/server/service did you use?

Local

Expected behavior

restic should ignore its own temp files during backup

Actual behavior

restic reports

error while saving data to the repo: remove C:\Users\USER\AppData\Local\Temp\restic-temp-pack-212416791: Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird.

Steps to reproduce the behavior

restic -r F:\Backup backup C:\Users\USER

Do you have any idea what may have caused this?

restic tries to backup its own temp file while restic of course is removing this file because it is temporary

Do you have an idea how to solve the issue?

restic should exclude its own temp files or temp directory from backup

[felix9]

I think this only happens on Windows. On Unix systems, restic uses unlinked temp files.

Workaround is to add the Temp dir to restic's excludes.

Plausible fixes:

1. Add os.TempDir to default excludes. I don't like this option because maybe you do want to backup other files in TempDir.
2. Keep track of temp files created, and exclude those. I don't like this option because multiple restic processes may try to archive each others' temp files. But that's arguably an esoteric case, and it's easy to workaround by changing temp dir.
3. Use a regular name pattern like "#restic.*" for restic temp files, and exclude any files that match that pattern. This of course will fail to archive files that accidentally or deliberately match that name pattern. Accident seems to me unlikely. But an antagonist evading archival is maybe a legitimate concern.
4. Just warn if the temp dir is in the archival set, and maybe try to automatically pick a temp dir outside the archival set.

[dirk777]

I would go for solution 2. I would not consider other restic processes. restic does not know of other processes at all, which is good. Creating for each restic process a unique temp dir should solve it. I am on your page with 1 and 3. The last solution won't work for the backup path C:\ and having only drive.

[fd0]

Thanks for the report and the analysis! I'd go with creating a subdir of os.TempDir() for each run of restic to place temp files in, and add this directory to the list of excludes.

[nicolas17]

A user on IRC reported this happened on Linux:

# restic backup /
enter password for repository:
scan [/]
scanned 89432 directories, 1320677 files in 1:43
error for tmp/restic-temp-pack-138128855: Lstat: lstat /tmp/restic-temp-pack-138128855: no such file or directory
error for tmp/restic-temp-pack-151660632: Lstat: lstat /tmp/restic-temp-pack-151660632: no such file or directory
error for tmp/restic-temp-pack-955784605: Lstat: lstat /tmp/restic-temp-pack-955784605: no such file or directory
[12:51] 9.02%  32.365 MiB/s  24.368 GiB / 270.058 GiB  69414 / 1410109 items  3 errors  ETA 2:09:33 

I guess this is super rare, it would be a race condition, if restic lists the contents of /tmp/ just after creating a temp pack but before unlinking it. But it can happen.

[fd0]

It's probably caused by a very old version of restic, on Linux, we've long since deleted temp files directly after creation so that this doesn't happen on Linux with recent versions

[fd0]

The user reported that they used restic 0.5.0, which still had this issue on Linux.

[nizedk]

I dont know if it's the case already, but if not, please also automatically exclude the path to the repository in use :-)

[fd0]

Yep, good point. Thanks!

[j6s]

Is this still an issue? Can't reproduce on Linux with 0.9.0

[fd0]

It is, although not on Linux, where restic unlinks the temp files directly after creating them (so they are not visible in the file system any more).

[Martin2112]

I could work on this but I sent a cleanup PR a couple of weeks ago and haven't had a response. Not sure how active this project is.

[MichaelEischer]

Review times for PRs vary between days and multiple months (unfortunately). That mostly depends on how complex a PR is, large PRs usually have to wait much longer until I find enough time to review them. And it of course depends on how much time I can spare for restic.

[Martin2112]

I've had a look at this. I don't think backing up the temp directory has any real value. The easiest option would just be to exclude os.TempDir() either just on Windows or on all OSes. However, if someone has TMPDIR set to a strange value this could affect what gets backed up. Does this matter? The alternative is, as discussed above, is to create a new directory and exclude that. I've had a go at this and it doesn't look too bad assuming I've got the right place where the files are created.

[rawtaz]

How come people don't just exclude the temp directory like they exclude other special directories that shouldn't be backed up? For example if you back up /, you naturally will exclude some paths, and I'd expect /tmp to be one of them. Similar approach on Windows of course.

If we automatically exclude one special directory, the question becomes to exclude other special directories too, no?

[MichaelEischer]

I've had a look at this. I don't think backing up the temp directory has any real value. The easiest option would just be to exclude os.TempDir() either just on Windows or on all OSes.

I'd prefer to move the temp files into a specific subfolder and then exclude only that.

[Martin2112]

There is an argument for doing nothing and let users do the excluding but the current behaviour is not consistent depending on which operating system is used.

[MichaelEischer]

There is an argument for doing nothing and let users do the excluding but the current behaviour is not consistent depending on which operating system is used.

I think it's a good idea to fix this issue. But I'd prefer to not make the decision for users to always exclude everything from the temp folder.