borg
borg copied to clipboard
borgbackup
Increasing (incremental) backup by tracking file changes in real-time?
Have you checked borgbackup docs, FAQ, and open Github issues?
Yes
Is this a BUG / ISSUE report or a QUESTION?
FEATURE REQUEST / Question
Describe the problem you're observing.
Borg takes a while to run an incremental backup, because it traverses the entire file system to compare file metadata with its cache. Has there been an attempt to track file changes in real-time, and then use that data in order to create the backup? I can see this being one of two possible methods of working (full file system traversal, and tracked-data backup). The idea would be:
Method 1: Using tracking data to speed up backup uploads
- Backup no. 1 at 12pm on Monday: upload dataset
- Between backup at 12pm Monday, and the next backup (12pm Tuesday), Borg runs a daemon in the background that tracks files which are modified and writes them to a cache
- Backup no. 2 at 12pm on Tuesday: uploads incremental backup, but instead of running through the whole file system, it simply backups up the files which have been modified
Method 2: Full file system traversal
This would be the same as it currently operates now, and the idea would be that once a week (or depending on whatever frequency you want to use), it would do the full file system traversal.
The idea here, ultimately, would be to make incremental backups faster. Every once in a while for a "sanity check" you could make sure to do a full traversal, in case somehow something got missed.
This would all be configurable in the settings, so some people would want full file system traversal all the time. Additionally, we would add a piece of metadata to each backup to indicate if it was done with tracking data vs. full file system traversal.
Borg needs to traverse and access all files, because it always creates backup archives which include all files and needs some metadata from all files.
There is no plan to transform borg into an incremental backup tool (like full + incremental or differential archives), because that is just painful to use (see other tools that do that).
If not much changed, backups are already quite fast with borg, so there's not that much to win here.
Thanks, @ThomasWaldmann. I agree that incremental backup tools are a different model. I just run it locally and find that with a very large backup set (even just things in my home folder, which includes ~/Library and lots of constantly-changing files there in particular), the sheer number of files and directories which are traversed make backups take 2-3 hours. I am thinking about how Borg might work better if that backup speed is reduced, potentially dramatically.
I have seen backups run quickly (and data transfer is very fast) when there are a small # of files, but when the # of files in the backup set gets larger and larger it just takes a long time to traverse all of that.
Btw I don't mean that each backup would be differential (e.g backup 1 is a full backup, then backup 2 is just the delta, etc.), but just as a way to track files as they are changed and then use that list as what to back up.
I have done this with a script before.
- Track the files however you want (I used any file newer than the last backup)
- Send those files to tar (dedup doesn't work AS good, but it does work)
- Pipe that tar output to borg.
Having borg do that internally is not a good idea IMHO. But if you arn't already, run borg locally, rather than targeting a remote host. 2-3 hours sounds like network latency/bandwidth killing your performance.
Won't piping tar output to borg just backup those changed files, not the whole backup set?
Also, running borg locally isn't always possible because you still need to store the data off of your computer.
Yes, and you would have to restore all increments in order. A pain and why there are other products dedicated to that.
Borg repos do not need to be local. There are multiple options from mounting a file system, to ssh, to Borg's own service model.
Just doing user@host:/path/to/repo is the easiest way though. If you have key pairs you don't even need to enter a password. I have used it very effectively even over an LTE connection.
@jalustig Something seems odd about your set-up if the backups are so slow. On my four-year old laptop, a backup to a remote server takes less than two minutes. Here are selected lines from a typical log:
Synchronizing chunks cache...
Archives: 381, w/ cached Idx: 381, w/ outdated Idx: 4, w/o cached Idx: 4.
Duration: 1 minutes 27.97 seconds
Number of files: 246448
Original size Compressed size Deduplicated size
This archive: 16.96 GB 10.55 GB 79.82 MB
This time includes the time to synchronize the chunks cache, since I backup multiple machines into the same repo. It takes 53s when run a second time immediately afterwards. So rather than trying to do complicated tracking of what has changed, I would investigate why borg is slow for you.
I think they are basically pulling the entire file structure over SSH to the backup server every time. So full read -> encrypt -> compress - > transmit -> decrypt -> decompress -> borg -> compress & dedup.
Ya, going to be slow as...
I'm running Borg on my local Mac and sending data over SSH to a server running on my local network.
Here is the results after running borg create:
Time (start): Fri, 2022-01-28 14:22:08
Time (end): Fri, 2022-01-28 14:53:46
Duration: 31 minutes 38.17 seconds
Number of files: 469217
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 294.24 GB 270.07 GB 1.70 GB
All archives: 13.29 TB 12.19 TB 807.33 GB
Unique chunks Total chunks
Chunk index: 927717 20183453
------------------------------------------------------------------------------
@jalustig Something seems wrong. You have around twice as many changed files as me, so it shouldn't take more than a couple of minutes to detect the changed files. (On the other hand, it's much better than the 2-3 hours you mentioned earlier.) Things to check: (1) Are you using a compiled msgpack? A faulty install will fall back on a pure python version, which is slow. There should be a warning if this is the case. (2) Does borg mistakenly think all of your files have changed? If you run borg with --list --filter=AME it will show you the files it thinks are Added or Modified (or have Errors). Maybe run borg on a subset while testing this. (3) Are you maybe using a very slow compression method?
You should post the exact command line you are using, the borg version, and any other relevant info. It would also be good to see the output from a run right after a previous run, so almost no files should be changed and the deduplicated size is close to 0.
@jdchristensen I don’t believe that it is seeing all files as “modified.” The command has the —list —filter=AME and I’m seeing that only some files are being listed as modified. It’s also a lot of files in ~/Library. I have a fairly extensive exclusion list to try to limit cache files being backed up but there are still a lot of temp files.
As for msgpack, how would I check this? I haven’t seen any error messages that would indicate this.
Is this from an HDD by chance? If you tar all the files to /dev/zero (not /dev/null as it will basically skip doing anything). How long does it take?
@BloodBlight this is on an SSD (the local computer). Data storage on the server is HDD.
@BloodBlight Finally got back on my computer and could do some testing. First of all, nowhere near all of the files have been modified, of course. Also: I ran pip3 list and msgpack was not listed.
So first, I ran borg create and this command and the output (see below)
Then, I installed msgpack (pip3 install -U msgpack)
Finally, I ran borg create again. I'll create another response with that (once it finishes) to keep them easier to tell apart.
FIRST TIME: (without msgpack)
➜ ~ borg create --exclude-from /Users/jason/.config/borg/exclusions --exclude-caches --exclude-if-present .nobackup --compression auto,lz4 --umask 2 --stats --debug --show-rc --progress --info --stats --list --filter=AME [email protected]:/storage/backup/borgbackup/jason::JasonsMacBookPro-documents-{now} /Library/Audio/Plug-Ins /Users/jason
Enter passphrase for key ssh://[email protected]/storage/backup/borgbackup/jason:
Creating archive at "[email protected]:/storage/backup/borgbackup/jason::JasonsMacBookPro-documents-{now}"
M /Users/jason/.config/borg/security/3e6888580fe1e8ace3fe090dfe8a87a46e342c19c8cdd49914d6019258f0dc0f/nonce
M /Users/jason/.config/borg/security/3e6888580fe1e8ace3fe090dfe8a87a46e342c19c8cdd49914d6019258f0dc0f/location
M /Users/jason/.config/borg/security/3e6888580fe1e8ace3fe090dfe8a87a46e342c19c8cdd49914d6019258f0dc0f/key-type
M /Users/jason/.config/borg/security/3e6888580fe1e8ace3fe090dfe8a87a46e342c19c8cdd49914d6019258f0dc0f/manifest-timestamp
M /Users/jason/Library/Application Support/AddressBook/Metadata/.infoation Support/AddressBook/Images/B82989D3-9C96-47B4-B685-AEC1B08FBF8B
M /Users/jason/Library/Application Support/AddressBook/Metadata/7FD6822F-6D5E-4C9D-82EB-C233344A1F0B:ABSmartGroup.abcdg
M /Users/jason/Library/Application Support/AddressBook/Metadata/EFCE06E8-C9EC-456A-93A7-494E3915AF63:ABSmartGroup.abcdg
M /Users/jason/Library/Application Support/AddressBook/Metadata/99F1DA75-A900-48F7-976C-C0DE8C353F03:ABSmartGroup.abcdg
M /Users/jason/Library/Application Support/AddressBook/Metadata/E63B97BD-6651-4F41-8B4A-8FB47429AAD5:ABInfo.abcdi
M /Users/jason/Library/Application Support/AddressBook/Metadata/8C63FFC2-D288-4999-9120-A1F9C3B60F3A:ABSmartGroup.abcdg
M /Users/jason/Library/Application Support/AddressBook/Sources/5C74EE72-A9C0-43CD-ACE3-BBDF6DC5FA8D/AddressBook-v22.abcddbBDF6DC5FA8D/AddressBook-v22.abcddb
M /Users/jason/Library/Application Support/AddressBook/Sources/5C74EE72-A9C0-43CD-ACE3-BBDF6DC5FA8D/Metadata/.info/EE1C7E80-5130-4CB7-A3AE-06191BC04F7F.jpeg
M /Users/jason/Library/Application Support/AddressBook/Sources/5C74EE72-A9C0-43CD-ACE3-BBDF6DC5FA8D/Metadata/4CCDB200-97E4-45DB-80DC-A5D68F061857:ABInfo.abcdi
M /Users/jason/Library/Application Support/AddressBook/Sources/5C74EE72-A9C0-43CD-ACE3-BBDF6DC5FA8D/Metadata/17E6F020-E70E-428E-9560-30F672EBF94C:ABGroup.abcdg
M /Users/jason/Library/Application Support/AddressBook/Sources/5C74EE72-A9C0-43CD-ACE3-BBDF6DC5FA8D/SyncAnchor56A-87EA-4527-9B43-8281F4FAADBD:ABPerson.abcdp
/Users/jason/Library/Application Support/Adobe/Lightroom CC Helper/lrcc_helper_exitstatus.txt: open: [Errno 13] Permission denied: '/Users/jason/Library/Application Support/Adobe/Lightroom CC Helper/lrcc_helper_exitstatus.txt'
E /Users/jason/Library/Application Support/Adobe/Lightroom CC Helper/lrcc_helper_exitstatus.txt
M /Users/jason/Library/Application Support/BetterTouchTool/Logs/com.hegenberg.BetterTouchTool 2022-01-27--06-54-19-248.log
M /Users/jason/Library/Application Support/BetterTouchTool/btt_user_variables.plistetterTouchTool/.dat.nosync7174.J8Mdn3
M /Users/jason/Library/Application Support/CallHistoryDB/CallHistory.storedata
M /Users/jason/Library/Application Support/CallHistoryTransactions/transactions.log
M /Users/jason/Library/Application Support/com.apple.sharedfilelist/com.apple.LSSharedFileList.RecentApplications.sfl2Documents/net.sourceforge.audacity.sfl
M /Users/jason/Library/Application Support/com.apple.spotlight/com.apple.spotlight.Shortcuts.v3
A /Users/jason/Library/Application Support/com.apple.spotlight/CandidateReports/com.apple.Spotlight/com.apple.Spotlight-2022-01-29.json
M /Users/jason/Library/Application Support/CrashReporter/Intervals_F14F4809-8F06-560E-AA21-7AAB8832EF6E.plistgent_F14F4809-8F06-560E-AA21-7AAB8832EF6E.plist
M /Users/jason/Library/Application Support/zoom.us/data/viper.inilication Support/TuneUp Uninstaller.app/Contents/Info.plist
M /Users/jason/Library/Application Support/zoom.us/data/zmonitorlog.bin
M /Users/jason/Library/Application Support/zoom.us/data/zoomus.enc.dbtion Support/zoom.us/data/[email protected]/giphy/3oEdva9BUHPIs2SkGk.gif
M /Users/jason/Library/Application Support/Knowledge/knowledgeC.dbplication Support/Knowledge/knowledgeC.db
M /Users/jason/Library/Assistant/CustomVocabulary/com.apple.iChat/0000000000000000000000000000000000000000/ContactGroupNameType/LatestVocabulary.plistdat
M /Users/jason/Library/Assistant/sync_flag
M /Users/jason/Library/Assistant/SiriAnalytics.db
M /Users/jason/Library/Assistant/com.apple.siri.corrections.laststate.plist
M /Users/jason/Library/Assistant/SpeechProfile_en_US
M /Users/jason/Library/Calendars/D0F228AD-B1D9-4D07-9545-29DC916F4E2F.caldav/321706F0-CB42-4B5C-8FFF-521B17F11941.calendar/Info.plistE-904D-DFEF9D71A8AB.ics
M /Users/jason/Library/Calendars/D0F228AD-B1D9-4D07-9545-29DC916F4E2F.caldav/63B201AB-4545-41F1-8B10-7DE5C53ADF5C.calendar/Info.plist
M /Users/jason/Library/Calendars/D0F228AD-B1D9-4D07-9545-29DC916F4E2F.caldav/67769446-A89C-4991-A165-037B646F66A6.calendar/Info.plist
M /Users/jason/Library/Calendars/D0F228AD-B1D9-4D07-9545-29DC916F4E2F.caldav/9C54206E-C535-4501-8B90-D64AAAC9FA69.calendar/Info.plist
M /Users/jason/Library/Calendars/D0F228AD-B1D9-4D07-9545-29DC916F4E2F.caldav/Info.plist
M /Users/jason/Library/Calendars/D0F228AD-B1D9-4D07-9545-29DC916F4E2F.caldav/5B548E5B-BD3E-455F-9ABD-07D8217A3EE8.calendar/Info.plistB-A8D2-A1761D70A6DD.ics
M /Users/jason/Library/Calendars/Calendar Cachers/jason/Library/Calendars/Calendar Cache
M /Users/jason/Library/Calendars/Calendar Cache-walason/Library/Calendars/Calendar Cache-wal
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/A0E37108-0F3D-4B6E-A821-1073A9621B0A.calendar/[email protected]
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/661774EF-8013-43DF-94A6-4217CCFCEF99.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/1B61E607-F4D8-4CF1-8F5C-D37CBE1FF953.calendar/[email protected]
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/BC3E2E19-EF3A-4786-BE80-8D9126B98412.calendar/Events/[email protected]
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/BC3E2E19-EF3A-4786-BE80-8D9126B98412.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/41DAC5A6-3C8A-45B9-8170-79C26558DCE8.calendar/[email protected]
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/452C6CF9-8105-4B73-B643-1982787237F4.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/E8B5B7E9-BF26-4DE8-B548-1C23DAFB9C1C.calendar/Info.plistA-AE8F-16D33005015D.ics
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/56C70615-1700-4ABF-B613-32C01C495FDE.calendar/[email protected]
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/1DD1DF52-4935-4284-9EE5-5BEE38776E9D.calendar/Info.plistB-BA67-7EBCCBDF16AC.ics
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/EC335A7F-79C0-4377-A5A3-BE8FCB30A6FD.calendar/[email protected]
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/57911490-DEFC-4E27-AF7B-C3BBDF88CF06.calendar/Info.plist October 5 - Staff Copy
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/1B0663BA-CFB6-4C90-87DB-A14091F78063.calendar/[email protected]
M /Users/jason/Library/com.apple.internal.ck/synched.db/Library/Calendars/NotificationIconCache/5-gregorian-en_US-1-8-1.png
M /Users/jason/Library/Containers/com.apple.CalendarAgent/Data/Library/Preferences/com.apple.CalendarAgent.plistFonts
M /Users/jason/Library/Containers/com.apple.corerecents.recentsd/Data/Library/SyncedPreferences/com.apple.kvs/ChangeTokens/EndToEndEncryption/recentsd/com.apple.mail.recents
M /Users/jason/Library/Containers/com.apple.FaceTime/Data/Library/Preferences/com.apple.imessage.bag.plistions/Data/Library/Audio
M /Users/jason/Library/Containers/com.apple.iBooksX/Data/CloudKit/2d041e2b0a4c1d15e9a908fecf6702280229ff30/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.iWork.Numbers/Data/CloudKit/03f7ccca0d107b4c9fb1c56113f7e0ada6d8bfa9/Records/pcs.dbtainers/TSAPersistentModel.db
M /Users/jason/Library/Containers/com.apple.iWork.Numbers/Data/CloudKit/ec7499b562f2205e09d363296e93ba298798ac74/Records/pcs.db
/Users/jason/Library/Containers/com.apple.mail 9.17.30 PM/Data/DataVaults: stat: [Errno 1] Operation not permitted: '/Users/jason/Library/Containers/com.apple.mail 9.17.30 PM/Data/DataVaults'
E /Users/jason/Library/Containers/com.apple.mail 9.17.30 PM/Data/DataVaults
M /Users/jason/Library/Containers/com.apple.Maps/Data/CloudKit/f03d436e26a625533de1873ce91bd8e20d58da1c/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.news/Data/CloudKit/5cbac0eccd05fa57626eae743234a76b342fc303/Records/pcs.dbtCollections
M /Users/jason/Library/Containers/com.apple.news/Data/CloudKit/98ba2b77c6c520b52bafd51ab6b92c38ff88c8f0/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.routined/Data/Library/Preferences/com.apple.routined.plistktop
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/19cd2669f37f65b31d5bbf4d0470072d0c2e24d2/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/19cd2669f37f65b31d5bbf4d0470072d0c2e24d2/Records/Records.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/301724c9d66005eb7ce7495bd423cc68b3f3ccb0/Records/Records.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/1ec5e976a8113dc85c2880c726a76ec1b7e8a855/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/dc2a6d1c798e4892c3b9b0d039dba980e2f44b18/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/11ff177eba84f9ea4295f0f10f672cc2bdd0d84b/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/30180fa649445281c39b0a47a7cd83134f5428fa/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/d2333c87bf94e084864d94a29ccb8df39544c2f9/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/Library/Preferences/com.apple.Safari.plist..33c87bf94e084864d94a29ccb8df39544c2f9/AssetsDb/Assets.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/Library/Safari/SafariTabs.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/Library/WebKit/WebsiteData/DeviceIdHashSalts/1/E41538F31106F0BC84F8FB4ECD63244A6EFAFAC2995FEFBFce__0
A /Users/jason/Library/Containers/com.apple.Safari/Data/Library/WebKit/WebsiteData/DeviceIdHashSalts/1/A114EAA64386C15C5F4C3D77474A3AD312F3777CDC46C9AB
M /Users/jason/Library/Containers/com.apple.mail/Data/Library/Preferences/com.apple.mail.plist/Li...Application Support/com.apple.spotlight/SubmittedReports
M /Users/jason/Library/Containers/com.apple.mail/Data/CloudKit/79648c1aa67564f3bb21117ed41e238cbafc40bc/Records/pcs.db2bf4fd028b18d17b01bf5ed127/MMCS
M /Users/jason/Library/Containers/com.apple.stocks.widget/Data/SystemData/com.apple.chrono/timelines/overview/systemMedium--6831538298694013486--344.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Containers/com.apple.stocks.widget/Data/CloudKit/6bece6a3d23379e43e34710331c4024e3bdde50a/Records/pcs.db
M /Users/jason/Library/Containers/com.apple.Photos.PhotosReliveWidget/Data/SystemData/com.apple.chrono/timelines/com.apple.Photos.PhotosReliveWidget/systemMedium--344.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Containers/com.apple.iCal.CalendarWidgetExtension/Data/SystemData/com.apple.chrono/timelines/com.apple.CalendarWidget.CalendarWidget/systemSmall---9068424113396816522--164.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Containers/com.apple.ScreenTimeWidgetApplication.ScreenTimeWidgetExtension/Data/SystemData/com.apple.chrono/timelines/com.apple.ScreenTimeWidgetApplication/systemMedium--344.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Containers/com.apple.weather.widget/Data/SystemData/com.apple.chrono/timelines/com.apple.weather/systemSmall--295258342008373080--164.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Containers/com.apple.MobileSMS/Data/Library/SyncedPreferences/com.apple.kvs/ChangeTokens/EndToEndEncryption/Messages/com.apple.messages.pinning
M /Users/jason/Library/Cookies/Cookies.binarycookiesson/Library/Cookies/com.apple.passd.binarycookies
M /Users/jason/Library/Group Containers/UBF8T346G9.ms/com.microsoft.autoupdate.fba/ceippref.plistunderkinder/Wunderlist/task-299313189.json
M /Users/jason/Library/Group Containers/UBF8T346G9.ms/Microsoft AutoUpdate.MERP.params.txt9.ms/MERP.Cxa.0EBBCEB5-8BA9-48AE-9039-881127D59200.plist
M /Users/jason/Library/Group Containers/UBF8T346G9.Office/MicrosoftRegistrationDB.regT346G9.Office/lckfe429ca1cf3fd702
M /Users/jason/Library/HomeKit/eventcounters-datastoren/Library/Group Containers/group.com.resilio.Sync/Library/Preferences/group.com.resilio.Sync.plist
M /Users/jason/Library/IdentityServices/ids.db
M /Users/jason/Library/IdentityServices/Persistence/personal-session-token-cache-v4
M /Users/jason/Library/IdentityServices/ids-pub-id.db
M /Users/jason/Library/IdentityServices/idstatuscache.plistrary/IdentityServices/incomingfiles
M /Users/jason/Library/Keychains/F14F4809-8F06-560E-AA21-7AAB8832EF6E/keychain-2.db8F06-560E-AA21-7AAB8832EF6E/keychain-2.db
M /Users/jason/Library/Keychains/F14F4809-8F06-560E-AA21-7AAB8832EF6E/Analytics/TransparencyAnalytics.db
M /Users/jason/Library/Keychains/login.keychain-dbjason/Library/Keychains/login.keychain-db.sb-827079bc-F867Cd
M /Users/jason/Library/LanguageModeling/en-dynamic.lm/dynamic.dat
M /Users/jason/Library/LanguageModeling/en-dynamic.lm/dynamic-lexicon.dat
M /Users/jason/Library/LanguageModeling/en-dynamic.lm/tags.dat
M /Users/jason/Library/LanguageModeling/en-dynamic.lm/meta.dat
M /Users/jason/Library/Messages/chat.db15 N Users/jason/Library/Messages/chat.db
M /Users/jason/Library/Messages/NickNameCache/nickNameKeyStore.db
M /Users/jason/Library/Messages/NickNameCache/handledNicknamesKeyStore.db
M /Users/jason/Library/Messages/NickNameCache/pendingNicknamesKeyStore.db
M /Users/jason/Library/Messages/com.apple.messages.geometrycache_v5.plist/NickNameCache/G_Gln2VVS+lZq+BmKe_0tQ==-ad
M /Users/jason/Library/Passes/PaymentWebServiceContext.archivery/News/com.apple.news.public-com.a...s.private-production/referralItems/com.apple.news.widget
M /Users/jason/Library/Passes/ScheduledActivities.archive
M /Users/jason/Library/Passes/RemoteDevices.archive
M /Users/jason/Library/Preferences/ByHost/com.apple.Bluetooth.F14F4809-8F06-560E-AA21-7AAB8832EF6E.plistF4809-8F06-560E-AA21-7AAB8832EF6E.plist
M /Users/jason/Library/Preferences/ByHost/com.apple.sharingd.F14F4809-8F06-560E-AA21-7AAB8832EF6E.plist
M /Users/jason/Library/Preferences/ByHost/com.apple.loginwindow.F14F4809-8F06-560E-AA21-7AAB8832EF6E.plist
M /Users/jason/Library/Preferences/Adobe/.963F69740A80
M /Users/jason/Library/Preferences/com.apple.remindd.plistibrary/Preferences/com.apple.security.sosaccount.plist
M /Users/jason/Library/Preferences/com.apple.ap.adprivacyd.plist
M /Users/jason/Library/Preferences/com.apple.appstored.plist
M /Users/jason/Library/Preferences/com.apple.cloudphotod.plist
M /Users/jason/Library/Preferences/com.apple.MobileSMS.plist
M /Users/jason/Library/Preferences/com.apple.parsecd.plist
M /Users/jason/Library/Preferences/com.apple.iChat.plist
M /Users/jason/Library/Preferences/com.apple.dock.plist
M /Users/jason/Library/Preferences/com.apple.imessage.bag.plist
M /Users/jason/Library/Preferences/com.apple.textInput.keyboardServices.textReplacement.plist
M /Users/jason/Library/Preferences/com.apple.internal.ck.plist
M /Users/jason/Library/Preferences/com.apple.facetime.bag.plist
M /Users/jason/Library/Preferences/com.apple.coreservices.useractivityd.dynamicuseractivites.plist
M /Users/jason/Library/Preferences/com.apple.IMCoreSpotlight.plist
M /Users/jason/Library/Preferences/com.apple.mmcs.plist
M /Users/jason/Library/Preferences/com.apple.madrid.plist
M /Users/jason/Library/Preferences/com.macpaw.CleanMyMac4.HealthMonitor.plist
M /Users/jason/Library/Preferences/com.apple.CallHistorySyncHelper.plist
M /Users/jason/Library/Preferences/com.apple.AddressBook.plist
M /Users/jason/Library/Preferences/com.apple.assistant.plist
M /Users/jason/Library/Preferences/com.apple.photos.shareddefaults.plist
M /Users/jason/Library/Preferences/knowledge-agent.plist
M /Users/jason/Library/Preferences/com.microsoft.autoupdate2.plist
M /Users/jason/Library/Preferences/com.apple.SafariBookmarksSyncAgent.plist
M /Users/jason/Library/Preferences/com.apple.spaces.plist
M /Users/jason/Library/Preferences/com.microsoft.autoupdate.fba.plist
M /Users/jason/Library/Preferences/com.apple.Spotlight.plist
M /Users/jason/Library/Preferences/com.apple.Terminal.plist
M /Users/jason/Library/Preferences/ContextStoreAgent.plist
M /Users/jason/Library/Preferences/com.apple.xpc.activity2.plist
M /Users/jason/Library/Preferences/com.hegenberg.BetterTouchTool.plist
M /Users/jason/Library/Preferences/com.apple.HIToolbox.plist
M /Users/jason/Library/RescueTime.com/log/debug_log.txtn/Library/Receipts/com.adobe.Acrobat.Pro/install.log
M /Users/jason/Library/RescueTime.com/rescuetimed.json
M /Users/jason/Library/Safari/History.db11 N Users/jason/Library/Safari/History.db
M /Users/jason/Library/Safari/Template Icons/CacheSettings.plist/Safari/Template Icons/5CB505321B41E80CBFAC23EBAC31C8A9.png
M /Users/jason/Library/Safari/CloudTabs.db
M /Users/jason/Library/Safari/SearchDescriptions.plist
M /Users/jason/Library/Safari/RecentlyClosedTabs.plist
M /Users/jason/Library/Safari/CloudHistoryRemoteConfiguration.plist
M /Users/jason/Library/Spelling/dynamic-counts.dat/jason/Library/Speech/Speakable Items/Application Speakable Items/Finder/Make a new Finder window
M /Users/jason/Library/Spelling/dynamic-text.dat
M /Users/jason/Library/Spelling/dynamic-text-tmp.dat
M /Users/jason/Library/SyncedPreferences/com.apple.syncedpreferences.plistferences/recentsd-com.apple.corerecents.map-locations.plist
/Users/jason/Library/VoiceTrigger/SAT: stat: [Errno 1] Operation not permitted: '/Users/jason/Library/VoiceTrigger/SAT'
E /Users/jason/Library/VoiceTrigger/SAT
M /Users/jason/Library/PersonalizationPortrait/PPSQLDatabase.dby/PersonalizationPortrait/PPSQLDatabase.db
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/INBOX.mbox/Info.plist3B-A070-...82C1B653D/Data/6/1/2/Attachments/216451/2.2/image001.jpg
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221643.emlx.emlxlx216107.emlxy A_.docc
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221644.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221645.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221646.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221648.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221649.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221651.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221652.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221653.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221654.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221655.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221657.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221530.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221650.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221659.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221661.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221662.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221663.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221664.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221613.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221605.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221619.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221621.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221640.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221665.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221666.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/Info.plist4762-88DA-CFB82C1B653D/Data/2/4/1/Attachments/142864/2
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/Drafts.mbox/Info.plistE-4762-88DA-CFB82C1B653D/Data/0/1/2/Messages/210670.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/Trash.mbox/Info.plistrash.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/9/8/1
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/Important.mbox/Info.plist
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/Receipts.mbox/Info.plistAD8C-5AD5F289973A/JSAN.mbox/Info.plist
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/.mboxCache.plist
A /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Inbox.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221647.emlxB82C1B653D/Data/1/2/2/Messages/221357.emlx
A /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Inbox.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221656.emlx
A /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Inbox.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221660.emlx
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Inbox.mbox/Info.plistFB-9D83-F519094E06C0/Inbox.m...1E-4762-88DA-CFB82C1B653D/Data/2/1/2/Attachments/212408/2/Halperin talk.jpg
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Archive.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Conversation History.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Deleted Items.mbox/Info.plistF519094E06C0/Deleted Items.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/5/9
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Drafts.mbox/Info.plistB-9D83-F519094E06C0/Drafts.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/3/1/2/Messages/213362.emlx
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Journal.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Junk Email.mbox/Info.plist83-F519094E06C0/Junk Email.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/6/8/1
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Outbox.mbox/Info.plist
A /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Sent Items.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221658.emlxFB82C1B653D/Data/1/1/2/Messages/211928.emlx
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Sent Items.mbox/Info.plist83-F519094E06C0/Sent Items.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/3/1/2/Attachments/213888
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Tasks.mbox/Info.plist
M /Users/jason/Library/Mail/V8/MailData/Envelope Indexon/Library/Mail/V8/MailData/Envelope Index
A /Users/jason/Library/Mail/V8/MailData/Envelope Index-wal
M /Users/jason/Library/Mail/V8/MailData/SmartMailboxesLocalProperties.plistilData/SmartMailboxesLocalProperties.plist
M /Users/jason/Library/Mail/V8/MailData/SyncedFilesInfo.plist
M /Users/jason/Library/Mail/V8/MailData/FlagMailboxes.plist
M /Users/jason/Library/Mail/V8/MailData/VIPMailboxes.plist
M /Users/jason/Library/com.apple.icloud.searchpartyd/ShareAttemptTracker/shareAttempts.plistyd/MasterBeacons/8B5978B9-5985-4E79-8AD3-DB38CB17DB03.record
M /Users/jason/Pictures/Photos Library.photoslibrary/private/com.apple.mediaanalysisd/MediaAnalysis/mediaanalysis.dbiaanalysisd/MediaAnalysis/mediaanalysis.db
M /Users/jason/Pictures/Photos Library.photoslibrary/resources/cpl/cloudsync.noindex/cloudphotos-1.0.plistpl/cloudsync.noindex/cpl_enabled_marker
M /Users/jason/Pictures/Photos Library.photoslibrary/resources/cpl/cloudsync.noindex/DownloadCounts.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/resources/cpl/cloudsync.noindex/syncstatus.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/resources/cpl/cloudsync.noindex/mobileCPL.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/resources/journals/Person-change.pljbrary/resources/journals/FileSystemVolume.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/resources/journals/Person.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/resources/journals/HistoryToken.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/database/Photos.sqlitebrary.photoslibrary/database/Photos.sqlite
M /Users/jason/Pictures/Photos Library.photoslibrary/database/search/psi.sqlitey.photoslibrary/database/search/psi.sqlite
M /Users/jason/Pictures/Photos Library.photoslibrary/database/search/zeroKeywords.data
M /Users/jason/Pictures/Photos Library.photoslibrary/database/search/searchProgress.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/database/search/graphDataProgress.plistry/database/search/graphDataProgress.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/database/search/searchSystemInfo.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/database/search/synonymsProcess.plist
M /Users/jason/.python_history Iconnd-attics/image00289.jpg
M /Users/jason/.zsh_sessions/_expiration_check_timestamp/.npm/_cacache/index-v5/b8/2a/c36dc76be58320d64ff3119ec9390122e6b9c71ad9dc1507dc9d9310c78f
A /Users/jason/.zsh_sessions/7F45F620-0CFB-4EFB-B6FF-3F36B43BE519.session
A /Users/jason/.zsh_sessions/AA48FCC2-AD91-443E-8F7E-A13B6D286907.session
A /Users/jason/.zsh_sessions/D41F6F59-B945-4F89-9D63-497079A8EAFC.session
M /Users/jason/.zsh_history.27 MB D 469126 N Users/jason/.gsutil/analytics-uuid
------------------------------------------------------------------------------
Archive name: JasonsMacBookPro-documents-2022-01-28T18:44:15
Archive fingerprint: a388f00e5e8b1ead83feef0c27af8f3c07bac7a38185b0e47eee2fd4c097a359
Time (start): Fri, 2022-01-28 18:44:24
Time (end): Fri, 2022-01-28 19:14:09
Duration: 29 minutes 44.23 seconds
Number of files: 469141
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 293.61 GB 269.98 GB 558.53 MB
All archives: 13.88 TB 12.73 TB 808.07 GB
Unique chunks Total chunks
Chunk index: 928758 21325061
------------------------------------------------------------------------------
terminating with warning status, rc 1
Hmm, the only thing that I can think of is you have some very large files that are only slightly changing, but borg has to re-read the entire file for only a few MBs of changes. But even on my old SATA SSD in my old server I am getting about the same "effective" speed (about 150MBps) in this situation.
I see a lot of .db files in there (sorry not a Mac user, so all guessing from here), but how big are those?
Maybe see what your largest files are:
find /Users/jason -type f -exec ls -S {} + | head -n10
Do those line up with the changes? I guess you put all those files in a script and add up all the space... But I am assuming you have fairly new hardware and should really be going faster.. I have this issue with a Minecraft server. One of the mods DBs is huge, but only has a few MBs of changes.
Also, I don't know what CPU you are using (Intel or M1), but I know my older Xeons REALLY like zstd over the others. I use it with zstd,11 and have very good results with speeds often over 100MBps on a 10 year old CPU. Some experiments on your hardware might yield improvements on both space and speed.
@BloodBlight - I'm not sure what is causing it to be slow. My only indication is that the process of reading the files is going slowly. My computer has a lot of small files (especially in the Mail directory, where apple's mail app stores each email as a separate file) and it just takes a long time to iterate through all of them.
EDIT: also to answer your question, this is running on a 2018 Intel MBP. So this is not a brand new computer, and it's not an M1, but this is no horse-and-buggy either.
Here is the second run through:
borg create --exclude-from /Users/jason/.config/borg/exclusions --exclude-caches --exclude-if-present .nobackup --compression auto,lz4 --umask 2 --stats --debug --show-rc --progress --info --stats --list --filter=AME [email protected]:/storage/backup/borgbackup/jason::JasonsMacBookPro-documents-{now} /Library/Audio/Plug➜ ~ borg create --exclude-from /Users/jason/.config/borg/exclusions --exclude-caches --exclude-if-present .nobackup --compression auto,lz4 --umask 2 --stats --debug --show-rc --progress --info --stats --list --filter=AME [email protected]:/storage/backup/borgbackup/jason::JasonsMacBookPro-documents-{now} /Library/Audio/Plug-Ins /Users/jason
Enter passphrase for key ssh://[email protected]/storage/backup/borgbackup/jason:
Creating archive at "[email protected]:/storage/backup/borgbackup/jason::JasonsMacBookPro-documents-{now}"
M /Users/jason/.config/borg/security/3e6888580fe1e8ace3fe090dfe8a87a46e342c19c8cdd49914d6019258f0dc0f/nonce
M /Users/jason/.config/borg/security/3e6888580fe1e8ace3fe090dfe8a87a46e342c19c8cdd49914d6019258f0dc0f/location
M /Users/jason/.config/borg/security/3e6888580fe1e8ace3fe090dfe8a87a46e342c19c8cdd49914d6019258f0dc0f/key-type
M /Users/jason/.config/borg/security/3e6888580fe1e8ace3fe090dfe8a87a46e342c19c8cdd49914d6019258f0dc0f/manifest-timestamp
/Users/jason/Library/Application Support/Adobe/Lightroom CC Helper/lrcc_helper_exitstatus.txt: open: [Errno 13] Permission denied: '/Users/jason/Library/Application Support/Adobe/Lightroom CC Helper/lrcc_helper_exitstatus.txt'
E /Users/jason/Library/Application Support/Adobe/Lightroom CC Helper/lrcc_helper_exitstatus.txt
M /Users/jason/Library/Application Support/BetterTouchTool/Logs/com.hegenberg.BetterTouchTool 2022-01-27--06-54-19-248.log
M /Users/jason/Library/Application Support/BetterTouchTool/btt_user_variables.plist
M /Users/jason/Library/Application Support/com.apple.touristd/bigsur_whats_new.icns
M /Users/jason/Library/Application Support/com.apple.touristd/catalina_macbook-pro_icon.icns
M /Users/jason/Library/Application Support/com.apple.touristd/mojave_macbook-pro_icon.icns
M /Users/jason/Library/Application Support/com.apple.touristd/macbook-pro.icns
M /Users/jason/Library/Application Support/com.apple.touristd/HighSierraBasics.icns
M /Users/jason/Library/Application Support/com.apple.touristd/com.apple.touristd.plist
M /Users/jason/Library/Application Support/Knowledge/knowledgeC.db/knowledgeC.db
M /Users/jason/Library/Calendars/Calendar Cache-walason/Lib...F7934.caldav/Inbox
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/A0E37108-0F3D-4B6E-A821-1073A9621B0A.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/661774EF-8013-43DF-94A6-4217CCFCEF99.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/1B61E607-F4D8-4CF1-8F5C-D37CBE1FF953.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/BC3E2E19-EF3A-4786-BE80-8D9126B98412.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/41DAC5A6-3C8A-45B9-8170-79C26558DCE8.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/452C6CF9-8105-4B73-B643-1982787237F4.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/E8B5B7E9-BF26-4DE8-B548-1C23DAFB9C1C.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/56C70615-1700-4ABF-B613-32C01C495FDE.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/1DD1DF52-4935-4284-9EE5-5BEE38776E9D.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/EC335A7F-79C0-4377-A5A3-BE8FCB30A6FD.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/57911490-DEFC-4E27-AF7B-C3BBDF88CF06.calendar/Info.plist
M /Users/jason/Library/Calendars/4BA2A897-F05F-468D-817B-84AE2FA400BE.caldav/1B0663BA-CFB6-4C90-87DB-A14091F78063.calendar/Info.plist
/Users/jason/Library/Containers/com.apple.mail 9.17.30 PM/Data/DataVaults: stat: [Errno 1] Operation not permitted: '/Users/jason/Library/Containers/com.apple.mail 9.17.30 PM/Data/DataVaults'
E /Users/jason/Library/Containers/com.apple.mail 9.17.30 PM/Data/DataVaults
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/19cd2669f37f65b31d5bbf4d0470072d0c2e24d2/Records/Records.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/CloudKit/301724c9d66005eb7ce7495bd423cc68b3f3ccb0/Records/Records.db
M /Users/jason/Library/Containers/com.apple.Safari/Data/Library/Preferences/com.apple.Safari.plist
M /Users/jason/Library/Containers/com.apple.Safari/Data/Library/Safari/SafariTabs.db
M /Users/jason/Library/Containers/com.apple.mail/Data/Library/Preferences/com.apple.mail.plist
M /Users/jason/Library/Containers/com.apple.stocks.widget/Data/SystemData/com.apple.chrono/timelines/overview/systemMedium--6831538298694013486--344.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Containers/com.apple.iCal.CalendarWidgetExtension/Data/SystemData/com.apple.chrono/timelines/com.apple.CalendarWidget.CalendarWidget/systemSmall---9068424113396816522--164.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Containers/com.apple.ScreenTimeWidgetApplication.ScreenTimeWidgetExtension/Data/SystemData/com.apple.chrono/timelines/com.apple.ScreenTimeWidgetApplication/systemMedium--344.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Containers/com.apple.weather.widget/Data/SystemData/com.apple.chrono/timelines/com.apple.weather/systemSmall--295258342008373080--164.00w--164.00h--20.00r.chrono-timeline
M /Users/jason/Library/Cookies/Cookies.binarycookiesson/Lib...tant.binarycookies
M /Users/jason/Library/Group Containers/UBF8T346G9.Office/MicrosoftRegistrationDB.reg
M /Users/jason/Library/HomeKit/eventcounters-datastoren/Lib.../datastore3.sqlite
M /Users/jason/Library/IdentityServices/ids.db
M /Users/jason/Library/IdentityServices/Persistence/personal-session-token-cache-v4
M /Users/jason/Library/IdentityServices/ids-pub-id.db
M /Users/jason/Library/IdentityServices/idstatuscache.plist...-84CB-9B56D783F55F
M /Users/jason/Library/LanguageModeling/en-dynamic.lm/dynamic.datos_analytics.db
M /Users/jason/Library/LanguageModeling/en-dynamic.lm/dynamic-lexicon.dat
M /Users/jason/Library/LanguageModeling/en-dynamic.lm/tags.dat
M /Users/jason/Library/LanguageModeling/en-dynamic.lm/meta.dat
M /Users/jason/Library/Preferences/ByHost/com.apple.sharingd.F14F4809-8F06-560E-AA21-7AAB8832EF6E.plist
M /Users/jason/Library/Preferences/com.apple.icloud.fmfd.plistudkeysyncing.plist
M /Users/jason/Library/Preferences/knowledge-agent.plist
M /Users/jason/Library/Preferences/com.apple.TelephonyUtilities.plist
M /Users/jason/Library/Preferences/com.apple.SafariBookmarksSyncAgent.plist
M /Users/jason/Library/Preferences/com.apple.xpc.activity2.plist
M /Users/jason/Library/Preferences/com.apple.Terminal.plist
M /Users/jason/Library/Preferences/ContextStoreAgent.plist
M /Users/jason/Library/Preferences/com.hegenberg.BetterTouchTool.plist
M /Users/jason/Library/Preferences/com.apple.HIToolbox.plist
M /Users/jason/Library/RescueTime.com/log/debug_log.txt/Lib...reaper.theme.plist
M /Users/jason/Library/RescueTime.com/rescuetimed.json
M /Users/jason/Library/Safari/Template Icons/CacheSettings.plist98935209439F.png
M /Users/jason/Library/Safari/CloudTabs.db
M /Users/jason/Library/Safari/SearchDescriptions.plist
M /Users/jason/Library/Safari/RecentlyClosedTabs.plist
M /Users/jason/Library/Spelling/dynamic-text.dats/jason/Lib.../Internet Explorer
M /Users/jason/Library/Spelling/dynamic-text-tmp.dat
M /Users/jason/Library/SyncedPreferences/com.apple.syncedpreferences.plistcagent
/Users/jason/Library/VoiceTrigger/SAT: stat: [Errno 1] Operation not permitted: '/Users/jason/Library/VoiceTrigger/SAT'
E /Users/jason/Library/VoiceTrigger/SAT
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/INBOX.mbox/Info.plist8
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221663.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221662.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221667.emlx
A /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/6D7A9FCA-381E-4762-88DA-CFB82C1B653D/Data/1/2/2/Messages/221668.emlx
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/All Mail.mbox/Info.plist
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/Drafts.mbox/Info.plist
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/Trash.mbox/Info.plist
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/[Gmail].mbox/Important.mbox/Info.plist
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/Finances.mbox/Info.plist
M /Users/jason/Library/Mail/V8/D4D959CE-A516-429A-AD8C-5AD5F289973A/.mboxCache.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Inbox.mbox/Info.plistx
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Archive.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Conversation History.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Deleted Items.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Drafts.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Journal.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Junk Email.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Outbox.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Sent Items.mbox/Info.plist
M /Users/jason/Library/Mail/V8/21CD1E4E-17B2-4BFB-9D83-F519094E06C0/Tasks.mbox/Info.plist
M /Users/jason/Library/Mail/V8/MailData/Envelope Indexn/Library/...MailData/Envelope Index
A /Users/jason/Library/Mail/V8/MailData/Envelope Index-walbrary/...Data/Envelope Index-wal
M /Users/jason/Library/Mail/V8/MailData/SmartMailboxesLocalProperties.plist
M /Users/jason/Library/Mail/V8/MailData/FlagMailboxes.plist
M /Users/jason/Library/Mail/V8/MailData/VIPMailboxes.plist
M /Users/jason/Pictures/Photos Library.photoslibrary/private/com.apple.mediaanalysisd/MediaAnalysis/mediaanalysis.db
M /Users/jason/.zsh_history91 MB D 469110 N Users/jason/pihole
------------------------------------------------------------------------------
Archive name: JasonsMacBookPro-documents-2022-01-28T19:17:05
Archive fingerprint: 58b89016b42812dd19fa0824c735a25663072e89c793f28e8081d028fe86f8f6
Time (start): Fri, 2022-01-28 19:17:28
Time (end): Fri, 2022-01-28 19:44:58
Duration: 27 minutes 30.34 seconds
Number of files: 469143
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 293.61 GB 269.98 GB 93.10 MB
All archives: 14.18 TB 13.00 TB 808.16 GB
Unique chunks Total chunks
Chunk index: 928952 21895587
------------------------------------------------------------------------------
terminating with warning status, rc 1
My understanding is, if the attributes of the file don't change. It shouldn't even open the file. So everything would just be meta ops, and the slowest of SSDs could do that for a half million files fairly quick...
I have a large directory at work that lives on a distributed file systems with well over a million files (and over a TB in size) that completes in about 10-50 minutes... So ya, something is up...
What does this return for you?
find ~/ -name "*.db" -exec du -b {} \; | awk '{total+=$1}END{print total}'
I have never used umask, does anyone know how much per-file overhead that adds? Not even sure what it does. :) @jdchristensen, do you know?
Umask affects the permissions of the storage container files on the server. I use it because of permissions weirdness. From what I understand it only affects the process of writing the Borg files at the destination.
I don't have any more ideas right now, except to be fairly sure that something unusual is happening. With only 93MB of new data and 500k files, it really should be much faster. Maybe @BloodBlight 's theory about large files needing to get rechunked is the issue? One thing you could do is run the backup on subsets of your filesystem and see if one subset is the culprit. You could do one run excluding Library, and then one only containing Library. If Library is the slow part, then try some of its subdirectories.
From the 1.1.16 change log entry:
- verbose files cache logging via --debug-topic=files_cache, #5659.
You could use that to find out why a lot of your files are marked as M.
@jalustig did you try my suggestion (see previous post) or did you otherwise solve the issue?