borgbackup
SMB Share WSL - Terrible performance
borg 1.2.8
borg create --info --files-cache=ctime,size --stats --progress --compression=lz4 "$BORG_REPO::$CURRENT_WEEKLY" $SOURCE_DIRS
Hello, I'm running borg 1.2.8 on WSL 2.6.1.0 to backup my C and T drives (Windows 11 24h2) to my NAS (TrueNas Scale) using Ethernet and a SMB Share Folder. My drives hold about 1.8 Million files (1,25 TB) and the first backup took 18 hour and 40 minutes, I thought that it was not too bad for the method used.
But right now I'm still having terrible performances after the inital backup. With some mesurement I'm finding a speed of about 50-60 files per seconds that are being analyzed. I've been doing a backup for 1h40 and 343296 files (117.13 GB) have been proccessed, only 2.89 GB of theses data are new.
I tried to search here and in other place what could slow down the proccess and I think that the simple answer is SMB share and WSL. I have a good and recent CPU that support virtualization and 32 GB of ram, I understand that this setup is a big bottle neck and I'm the one to blame for that but is there really no solution to make the process more efficient and waste less time with the SMB or the Mount ?
And if there is no easy solution to this, what would you recommend as a better solution to create my backups ? (including the migration steps)
Thanks you very much for thoses who will read and answer to my question.
please check the faq, there are instructions about how to debug and resolve speed issues.
@ThomasWaldmann Thanks for the response. I did check the faq but didn't find a solution. perhaps I missed it ? "Why is backup slow for me?" : My files are not modified/added, I have a only few Gb of change so that's not the issue. the VM i'm using, WSL, is using hardware virtualization so speed should be okay on that side.
So, what status char (leftmost in log output) does it show for most files when using borg create --list ...?
here are the log files of 2 test run I did. There are a bunch of updated files there but not all of them. the prexisting files still take a lot of time
So the files are all "M"(modified), "A"(added) (and a few "E"(error) due to permissions).
That means that borg thinks they are modified and that is why you get bad performance.
Read and follow the FAQ to resolve this.
https://borgbackup.readthedocs.io/en/1.4.2/faq.html#why-is-backup-slow-for-me
but... what ? are you sure ? I read that already but they are not all "M" and "A" ? I see a fair quantity of M and A but if you look at test4.log for example, nearly all file between 18.41 GB and 49.63 GB dosen't show "M" or"A" and it still took a really long time. Monitoring my Ethernet usage I can also see that the vaste majority of my exchange bitrate is arond 12-24KB, it spikes to around 100MB when file transfer is actually happening.
That log is a mess because you mixed --progress and --list.
For more clarity, just use --list without --progress.
And likely your smb server messes up ctime somehow.
Maybe it works better with --files-cache=mtime,size. You need to run at least 1 complete (and slow) backup before you will see an improvement in the next backup.
I see now. I tried --list without --progress and it seemed to be mostly "U". I will try to do a full backup with --files-cache=mtime,size and see how it works on the second backup. I will close the issue if what you suggested end up working, Thank you very much for the advices and your patience 🙏🏾
Waiting for next opportunity to do a backup but it didn't seemed to work. I Did a full back up, then re did it just after and it was still around the same speed
I'm still under the impression that the problem here is SMB Share and/or WSL mount for the source AND The destination directory, and if it is the case, what would be a better solution ? Originally I didn't try the windows version of BorgBackup because I heard it lacked proper support, but just now I'm wondering if that would be a better solution and what would be the drawbacks to that.
You said you see mostly "U". That should solve the speed problem. "U" means that borg does not read the files' content because it is unchanged. It then takes the chunk ids from the files cache to create a backup item for that file, which is very fast.
I can't say much about windows usage, I personally don't use it regularly and there is no real "borg windows maintainer".
You said you see mostly "U". That should solve the speed problem. "U" means that borg does not read the files' content because it is unchanged. It then takes the chunk ids from the files cache to create a backup item for that file, which is very fast.
Sure, but maybe that simply listing the files and their metadatas is too slow for my setup? I don't know, just trying to find an explanation. I'll try the windows version when I'll be back where my NAS is and see if I can improve speed there. Hopefully I will find a solution for this problem and thanks for your help.
for future reference, I got confused, the WSL method IS the Windows verison and is experimental