borg
borg copied to clipboard
Published
20 hours ago •
borgbackup
borgbackup
Integrity error
Have you checked borgbackup docs, FAQ, and open GitHub issues?
Yes
Is this a BUG / ISSUE report or a QUESTION?
Bug
System information. For client/server mode post info for both machines.
Borg used here:
https://github.com/nextcloud/all-in-one/issues/3442
Your borg version (borg -V).
1.2.6
Operating system (distribution) and version.
5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
Hardware / network configuration, and filesystems used.
Used nextcloud-aio for backup
Backups are done via docker, to the NFS folder. Storage is hard drive connected to Raspberry Pi 4 with ubuntu 22.04. Maybe worth noting - yesterday I did sudo apt update && sudo apt full-upgrade -y there
How much data is handled by borg?
~35Mb
Full borg commandline that lead to the problem (leave away excludes and passwords)
sys.argv: ['/usr/bin/borg', 'create', '-v', '--stats', '--compression', 'auto,zstd', '--exclude-caches', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/nextcloud.log*', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/audit.log', '/mnt/borgbackup/borg::20230929_050451-nextcloud-aio', '/nextcloud_aio_volumes/']
Describe the problem you're observing.
Integrity error when creating backup
Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.
Yes, run backup again. Tried repairing with nextcloud-aio but to no avail
Include any warning/errors/backtraces from the system logs
Problem appeared during today's backup: https://github.com/nextcloud/all-in-one/issues/3442
Copying logs here: Backup folder is an NFS mount which was not used by anybody. Below are backup logs:
Performing backup...
Starting the backup...
Creating archive at "/mnt/borgbackup/borg::20230929_050451-nextcloud-aio"
Data integrity error: Invalid segment entry header [segment 37, offset 27231398]: unpack requires a buffer of 41 bytes
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main
exit_code = archiver.run(args)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run
return set_ec(func(args))
^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
return method(self, args, repository=repository, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 664, in do_create
create_inner(archive, cache, fso)
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 631, in create_inner
str(cache),
^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/cache.py", line 418, in __str__
return self.str_format.format(self.format_tuple())
^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/cache.py", line 445, in format_tuple
stats = self.stats()
^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/cache.py", line 435, in stats
archive = Archive(self.repository, self.key, self.manifest, archive_name,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archive.py", line 488, in __init__
self.load(info.id)
File "/usr/lib/python3.11/site-packages/borg/archive.py", line 501, in load
self.metadata = self._load_meta(self.id)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archive.py", line 491, in _load_meta
data = self.key.decrypt(id, self.repository.get(id))
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1205, in get
return self.io.read(segment, offset, id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1609, in read
size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ), read_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1624, in _read
raise IntegrityError('Invalid segment entry header [segment {}, offset {}]: {}'.format(
borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment entry header [segment 37, offset 27231398]: unpack requires a buffer of 41 bytes
Platform: Linux cd8fe7ce0b70 5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64
Linux: Unknown Linux
Borg: 1.2.6 Python: CPython 3.11.5 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 36 CWD: /
sys.argv: ['/usr/bin/borg', 'create', '-v', '--stats', '--compression', 'auto,zstd', '--exclude-caches', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/nextcloud.log*', '--exclude', '/nextcloud_aio_volumes/nextcloud_aio_nextcloud/data/audit.log', '/mnt/borgbackup/borg::20230929_050451-nextcloud-aio', '/nextcloud_aio_volumes/']
SSH_ORIGINAL_COMMAND: None
Deleting the failed backup archive...
Data integrity error: Invalid segment entry header [segment 37, offset 26104925]: unpack requires a buffer of 41 bytes
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main
exit_code = archiver.run(args)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run
return set_ec(func(args))
^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
return method(self, args, repository=repository, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 1187, in do_delete
return self._delete_archives(args, repository)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 1259, in _delete_archives
archive.delete(stats, progress=args.progress, forced=args.forced)
File "/usr/lib/python3.11/site-packages/borg/archive.py", line 1011, in delete
for (i, (items_id, data)) in enumerate(zip(items_ids, self.repository.get_many(items_ids))):
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1211, in get_many
yield self.get(id_)
^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1205, in get
return self.io.read(segment, offset, id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1609, in read
size, tag, key, data = self._read(fd, self.put_header_fmt, header, segment, offset, (TAG_PUT, ), read_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1624, in _read
raise IntegrityError('Invalid segment entry header [segment {}, offset {}]: {}'.format(
borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment entry header [segment 37, offset 26104925]: unpack requires a buffer of 41 bytes
Platform: Linux cd8fe7ce0b70 5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64
Linux: Unknown Linux
Borg: 1.2.6 Python: CPython 3.11.5 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 37 CWD: /
sys.argv: ['/usr/bin/borg', 'delete', '--stats', '/mnt/borgbackup/borg::20230929_050451-nextcloud-aio']
SSH_ORIGINAL_COMMAND: None
Backup failed!
You might want to check the backup integrity via the AIO interface.
As well as integrity check logs:
Checking the backup integrity...
Starting repository check
Data integrity error: Invalid segment magic [segment 37, offset 0]
finished segment check at segment 131
Starting repository index check
Index object count mismatch.
committed index: 38224 objects
rebuilt index: 38071 objects
ID: 900a7f0ae24982b935cf5fde8ffe4fb0a701b89e7f6080f85235071c4ccd6c42 rebuilt index: <not found> committed index: (37, 17018617)
ID: 2e12b30e6223b7365aecdc4e3049e9fefbcbe1aa6349729c2ff0e128805df668 rebuilt index: <not found> committed index: (37, 9494502)
ID: b04efa4a611343895cd6d81994bc6336cb810331141b792b6c5d93518b240c6f rebuilt index: <not found> committed index: (37, 16262069)
ID: 734cde44dc7f5ac637b9a62033dbae8f6996dda1ce1d76a2621d271a02b75680 rebuilt index: <not found> committed index: (37, 9461290)
ID: d12410184ae062a821e98d51b71abab451db2680f8220ea368bd3c5833a6cd9f rebuilt index: <not found> committed index: (37, 16476660)
ID: f435cb25283e44b4a677c9d69cc80ce30bbbc38c86a1cb089f3eae5d632fc0a3 rebuilt index: <not found> committed index: (37, 7061023)
ID: a4d319c284e37b422ce34483774eda8f4b4bcba6570a8c8a0f1b427c27b542ec rebuilt index: <not found> committed index: (37, 16524458)
ID: 27c7e7b4709bef1ad425d1f207c9999e3c1dd0d5b229374ee9d61a0429069a18 rebuilt index: <not found> committed index: (37, 17082837)
ID: ed25ab115820ab5bb5219c3b1ef5839d0317b2c7281eda716b3fceb59be40cdf rebuilt index: <not found> committed index: (37, 3975673)
ID: 257fbf69d84ee8d65f8e4c618ccf696805799da642e152baec3fe195a34076ae rebuilt index: <not found> committed index: (37, 6419533)
ID: 48208509a3b932fb3af66c6955d6739a12fbe35ebb6c8d82df8b199a0c408b76 rebuilt index: <not found> committed index: (37, 16833543)
ID: f461164b8fad1ba4bd210b26a720923a8ee9fd49c417c32f66014ad3253757ca rebuilt index: <not found> committed index: (37, 17082257)
ID: 47d6a1bd288cd092b28ba058fcc017a456cd43231b7f3e7df4322f8f94b81fbe rebuilt index: <not found> committed index: (37, 17055146)
ID: cd840c6ba4bfc218923ddc86e93a0e74b5c9a88f8cd268bee1d26e6d9e4f1c44 rebuilt index: <not found> committed index: (37, 17008824)
ID: e197767a4e8c7566d59891887f202330a847d6f76a5d04af326b66c2f3c93b0c rebuilt index: <not found> committed index: (37, 17231783)
ID: 87760d590eb5d43f8a87ca80af2cb2b2c007aa4b05e6b01c143cb756647854eb rebuilt index: <not found> committed index: (37, 9445159)
ID: dc1205f20f9edbde210a122721785c3cbd807d3fd467aa7b044ba270c825ddb4 rebuilt index: <not found> committed index: (37, 26443328)
ID: 8197b775cee0823bc6200bd5fb480be79792d21848e02f40eea6c971bd9213b2 rebuilt index: <not found> committed index: (37, 27133011)
ID: 6a504a2c3af9b9b87835d9c0512b516c5c4436cf99de3e8d52003e902caf2461 rebuilt index: <not found> committed index: (37, 3385516)
ID: c9a14a7c6f14ab937391b07dcf09c0442cf8bf30ee3b9df9f465850df8d5df91 rebuilt index: <not found> committed index: (37, 9477394)
ID: 4d59b53172b47694b850f98a44b3da6f44432645962a6ea3a7343032a7e775d0 rebuilt index: <not found> committed index: (37, 9713319)
ID: 00ba3e91e751318ae94602a1a4f2d3d034300c5d8de13ffeaece166f45fae85a rebuilt index: <not found> committed index: (37, 10963984)
ID: 6195c06a28e6b7a01ff9827333f4c1448a812dd13321891c358606f448902af9 rebuilt index: <not found> committed index: (37, 9707093)
ID: 75e201b76ea986ed2f5e53b56096ca635955c3083a239cac587008b63b40fa08 rebuilt index: <not found> committed index: (37, 16900451)
ID: 9de79ebb49ce3b811241b9a79f3736f51a7e00f4cd9718442e89d4149e86cb5e rebuilt index: <not found> committed index: (37, 12912109)
ID: 8f4be81eb85f7c157cf6628bad9367705c5d8011378e3f9936e9d9468da72bab rebuilt index: <not found> committed index: (37, 17081613)
ID: 1ebfc691ff8bb2c20b884c9acb8e60f96a43657a316143960d5297a72e175b11 rebuilt index: <not found> committed index: (37, 17232164)
ID: 9fb7ce87f15a8a5740eab922e0f4aefa656254aff0b50d5329e5dcc88f624b6c rebuilt index: <not found> committed index: (37, 17064718)
ID: 28654c34d87fe83194e2dc7b060afcac989672d8c56b81cae65157e37dee3fea rebuilt index: <not found> committed index: (37, 26194750)
ID: fcbf9189727768b21205b71a90e43b327986ff64993835e4d197051735642e6a rebuilt index: <not found> committed index: (37, 17080011)
ID: 40d48a9cbd22059fa75fe896798c64ee74ec8235f8247cc606b311a201af60fd rebuilt index: <not found> committed index: (37, 5263819)
ID: eabfe582aedc9c4bcb95a5c66bef6bd911178044ff61517c3e8280ee9b6330ba rebuilt index: <not found> committed index: (37, 16433021)
ID: cf11cad4045d0c6272c4ebaeda070f096ebc7aa46af9ff36eb77af85101bc865 rebuilt index: <not found> committed index: (37, 9494133)
ID: d67e4240c04ed2714db4a01bffaa7fc7352d6ceac197626d82458e1446bb477d rebuilt index: <not found> committed index: (37, 15763325)
ID: 3abea47e09e1f45acf21747bf52c089e58f37ede37956fc33e07f486f7fdd33d rebuilt index: <not found> committed index: (37, 2168690)
ID: a871dd31fee63ca02b82464bb658fb4464d4b6958e29d947917fb0a29ccafa9b rebuilt index: <not found> committed index: (37, 9493916)
ID: 15434b033bffaff476af338102df48f141aa0ae78c4337c8652e5329f64ac62b rebuilt index: <not found> committed index: (37, 16425226)
ID: a1957f54eae59abc18d054bf550905f1c81f6aeea18c70eb2f763087add01ff0 rebuilt index: <not found> committed index: (37, 16500516)
ID: fa6b802acff6270f6869a88955a1a6cdb572504044164a329484ae6d55a24752 rebuilt index: <not found> committed index: (37, 2967068)
ID: b4e610a5df155ec39a4b5dd4fd626d88586662071e3f03fdb8e4ac570b05d179 rebuilt index: <not found> committed index: (37, 17070133)
ID: 9ee69ca499d2f66bf9dad87010d0a2d4880850fe82c30b02f4a29dd15b260bba rebuilt index: <not found> committed index: (37, 16643481)
ID: 5b2538e25c573484b54f1c6c0f4b033f8a0f0d210479fead197c580424a114aa rebuilt index: <not found> committed index: (37, 17214327)
ID: 5ca3225fd701c457d7c39331ec2ffcc4b537e85a1761397c05b86214267f75f4 rebuilt index: <not found> committed index: (37, 16282362)
ID: 4f3d87f779be696d08c3a1ea5302c9ab6a1c39e05ab08680d3f7a85c80c55418 rebuilt index: <not found> committed index: (37, 27231398)
ID: 6339f5f28776adaafe6279e29ce41c3a546022f36d56331a10aad3721cce73d6 rebuilt index: <not found> committed index: (37, 17085847)
ID: 019df05211d00f78e7e9409aa61927435099c4342a87f29a21ec9b323acf37f4 rebuilt index: <not found> committed index: (37, 16959380)
ID: 188a8c3fb3ba56a38c3a5c4619f8009a4da1c27d68f7dcadbbe02a28c3525d19 rebuilt index: <not found> committed index: (37, 15357515)
ID: dad9c08c06bc37c9e2a8296cde2d9fc68d73507502e64dc0f3f989a21b07eef3 rebuilt index: <not found> committed index: (37, 17078405)
ID: 1c57cc073c97e42366449750a7d308231e5f429b94cafd791080e1f8bae46e50 rebuilt index: <not found> committed index: (37, 9706960)
ID: 5617b2c62057eb15c85ee300e0224678ca2a7609f7fd73472391980a83a463a6 rebuilt index: <not found> committed index: (37, 8)
ID: ef3724e557a4e0d8124d317c0ec04af5c362756ac79f56a7e2888ca5b89e0836 rebuilt index: <not found> committed index: (37, 9493649)
ID: 603efeea83c287d5a4b3ab34ecea3c976498ce41a361fce2a97ac261ee9d9ae2 rebuilt index: <not found> committed index: (37, 16268825)
ID: 5b93c43f0706f978d7e3bf7e1911074aaa17d2a96860efa54de7472c8fb37c0f rebuilt index: <not found> committed index: (37, 16251134)
ID: b021dec956e4579fdb3a20be34fac781a0d34dfaadf825aa52a7a53db7945573 rebuilt index: <not found> committed index: (37, 17018780)
ID: ff32addae50d4a0eb451fdc12eac4f6f73b82715c5edd4e5dfc9fa1a8d1b001c rebuilt index: <not found> committed index: (37, 197921)
ID: 10ea858c7bba54ca0b99b434c53a4b2cd6f3c6eea4ed411660614af8b0e9f165 rebuilt index: <not found> committed index: (37, 1158)
ID: d0a4d5404b8552970f77fefa40bd5bc9eb9c4c9bcc434c10936a4a31b6190ba3 rebuilt index: <not found> committed index: (37, 26048511)
ID: def6bd90cb4f515ff28feaf27dd329663aebd34881dd66425465c08c9d69af20 rebuilt index: <not found> committed index: (37, 24171689)
ID: 7d148facf73666ba0f49bb5366c444976b4ab85457610edb8425ab97320feb19 rebuilt index: <not found> committed index: (37, 17038635)
ID: b2e77c7f2ef62b47f31ec3ede3eb4932079c5fbc38f2d70a596c34037707d982 rebuilt index: <not found> committed index: (37, 16957997)
ID: ba75330c03d04b0b6603e93ec318b6daa1eaaad7001c04f443887a0a5b4df93f rebuilt index: <not found> committed index: (37, 9495446)
ID: 6807999ac5372c90f77cb44a83018cc4485bbc014eb404a8216f84b84cb61922 rebuilt index: <not found> committed index: (37, 17034605)
ID: e32237b41a12a2775c065802cede8656001fbfe39e7661804a080ca73555cad7 rebuilt index: <not found> committed index: (37, 16961256)
ID: 95b8904386d2b5c7ead72b11400d4e270ca918d2bd7be39e47a512839384ed6e rebuilt index: <not found> committed index: (37, 15956754)
ID: bd1fe0a93a316beafd7f5d629914dd830f48212891842049a19a43bb8c633c45 rebuilt index: <not found> committed index: (37, 14459160)
ID: 7587521046d95beb718c1cc3709a1d4d008a047c317b979419a4eb6003181ce0 rebuilt index: <not found> committed index: (37, 9494629)
ID: 01909516c8d7ed549c99255a20001ee53c712c794a7102c7b6d140e61fa0094c rebuilt index: <not found> committed index: (37, 15310984)
ID: 2ee7236ba0aa3a176f882b9e02ba0e8cfa1395573004ea7710829e0d0cd3c75d rebuilt index: <not found> committed index: (37, 6818677)
ID: 972a50adc08a597f56eb88cb9906255071f12150be8b2f7c27952e761110210e rebuilt index: <not found> committed index: (37, 8559185)
ID: 820f2790cedfeff99d0c52a33051cd4490a7784175b8ed7ff8f3e95a658cea9a rebuilt index: <not found> committed index: (37, 9495320)
ID: 13435ac3ac2a2b69f81bd01bbe6d0a92cfed1c5d6a3708109069c9a3167c13c6 rebuilt index: <not found> committed index: (37, 17214125)
ID: 8aae502e22cbaa3b11305c88b645ae3a5384e4b4e3e361053687c1859dda1545 rebuilt index: <not found> committed index: (37, 17084118)
ID: c54db3cc78b07acaa6cd3684cc91c5cf2341e804f6943fc202a294800d19c133 rebuilt index: <not found> committed index: (37, 16959824)
ID: 4929dda7956a86b0f255e14c106a8d3e1f92ec4743374dbbf4bb53af4652b1e2 rebuilt index: <not found> committed index: (37, 25171390)
ID: f01b699929fe564d744c018cbcf7bdadda17e4dc492fe1a21e865bca3abeaed3 rebuilt index: <not found> committed index: (37, 26085602)
ID: 74024f7d751bdb93ae94ce0b89480416302a490558ba53b2e161990149c2888d rebuilt index: <not found> committed index: (37, 16424242)
ID: 15ee9d68baaebf59d7d79c23b0afdebc3290e36f3a654d75e8e003bef757ec37 rebuilt index: <not found> committed index: (37, 10644452)
ID: 41e36c5d0835a9987a8115157f07e4e9299eaa540a3a40230d06995ff7cd0940 rebuilt index: <not found> committed index: (37, 9494255)
ID: 7d34abadd06971c942ed79f5fbd9f7b106aed3c9d4850c79f567eed64a34ca47 rebuilt index: <not found> committed index: (37, 969)
ID: ba8094f7de0c9b84e6379045d5493586735be5bf27c0e15d2971282fc24a2c45 rebuilt index: <not found> committed index: (37, 3608993)
ID: 471fb5953f181883da92ab7634340078f0dffb18eb3ff27b9256150b4f354af7 rebuilt index: <not found> committed index: (37, 16251295)
ID: ca1c9f927876ef21555f4b0d45ab2a430a96f80b4411d22e0673398e6ff3bfc2 rebuilt index: <not found> committed index: (37, 17017291)
ID: 33cb8e3fe6dc54e7f45c7042ad6168f39df0702c67781e42429c6d1b8d2341ea rebuilt index: <not found> committed index: (37, 1344727)
ID: 6a228c96a54f88da96826b6f676ea662ee2ebc003a07cebcc4b26c2ef3dcda5b rebuilt index: <not found> committed index: (37, 13086943)
ID: a0d9414ca1d53dbce17fd508fe14de7bc5283f4fbe4fc5d85524b34017315ddc rebuilt index: <not found> committed index: (37, 9741743)
ID: c6f32562cfdf9f578b46ac3d6cd47edfefe0450fd0ae36f8ff652b25434826cd rebuilt index: <not found> committed index: (37, 26104925)
ID: 5535eba213a3a658907a1f8bdb5677677d634d3a7e64713f72ba7a0543df360b rebuilt index: <not found> committed index: (37, 20484967)
ID: b75287bc5aac80f8b386ad8a089e971d005315e0f14f0bfe60a87f95326855ad rebuilt index: <not found> committed index: (37, 27192386)
ID: a39a62023968c8b4c2ee1eabe8faf9d25ad0b795bcf47bf3c054247784bf5e91 rebuilt index: <not found> committed index: (37, 2308439)
ID: 5b27fc8d663acc7fa40735b812be9b4a94f7910f4162849adf6296a6095c85e2 rebuilt index: <not found> committed index: (37, 16430841)
ID: 078800e71d4dfaeeb4090b55aac23d255cbf63c80cc5be1badf97d47f1fb6090 rebuilt index: <not found> committed index: (37, 9493516)
ID: 3d24d8807fd39298c1f92d6a20a4dae22d76f86132e5afbf95dffca7093d5288 rebuilt index: <not found> committed index: (37, 8595235)
ID: e8344591eaba9d2f384ef99a73ca732ae4970e57cebbabc0530c2af54d03af54 rebuilt index: <not found> committed index: (37, 12240764)
ID: 4b8b98e6a992b249b5615423d228aa72210b7e9e67c0cda78a8546fa02564b53 rebuilt index: <not found> committed index: (37, 16264695)
ID: b63ef698833b0d7c2354b0d51ca25b026672f70e9bb6f0c7a9fc042b8ca50d4f rebuilt index: <not found> committed index: (37, 3722799)
ID: 8a34958b84f4ca3a8c2162a579fe83e8bd1b4ab58b35725383d97bf4ecbd35e7 rebuilt index: <not found> committed index: (37, 16266736)
ID: ae0953609bcb8aea24a6b8869b0bdbdff3b9f2ed6c7a21e46aa7e2f35272420c rebuilt index: <not found> committed index: (37, 26117233)
ID: d4c4a61a76ab23c16b2bd3f0ea7f5e582bc2cec31f34d057f3c12abd73ae4422 rebuilt index: <not found> committed index: (37, 9494994)
ID: 6a360f87885f2bc212d8f4dfea8d0020111eea286a2f0bfe6cc30cfeab3cd639 rebuilt index: <not found> committed index: (37, 17156158)
ID: 629d70e970dcb0080e3d3761f7005a9fd3341bc381770778634f143bfc185781 rebuilt index: <not found> committed index: (37, 16281140)
ID: 0d6e1abae7ab0802035900a9f3a7cd2b72e86bdebbd73340b20e0da6d72e26bb rebuilt index: <not found> committed index: (37, 16282525)
ID: efde462a5cca7505d51c9ee09bd89357d693cbcd8000259e46fdd1d5e8f95f5c rebuilt index: <not found> committed index: (37, 16811560)
ID: f77774c27d1f68299636bd70681d5813797c7aff6cc4904d6e5770963a54d950 rebuilt index: <not found> committed index: (37, 16548302)
ID: 91b8a601153815b4566918eed586af041b888a3b1dbb97b7917d12045c6bf426 rebuilt index: <not found> committed index: (37, 16247888)
ID: 4420dc671dc4de910a0e0528b249be46013712ccb0f114d04034f5b276dace86 rebuilt index: <not found> committed index: (37, 14521479)
ID: a8448d8754c70df8932056eac7d5b2d9466ab087ea8f80564ee1087d4e82776a rebuilt index: <not found> committed index: (37, 4662582)
ID: 4309e84b4ba2f956b5eac29fd381388378ce6bc9ff9df86b5c049e4d77fbbe44 rebuilt index: <not found> committed index: (37, 18201277)
ID: 36fef83ff8a09636c548891eac27945c4b110a86e6d01e981ef5e83ad42a7ae6 rebuilt index: <not found> committed index: (37, 7480172)
ID: ee1c185e6232d51bd9be52751e2934f03b3433b157523755b67af5a898aa669e rebuilt index: <not found> committed index: (37, 11296900)
ID: dd6683a71374cba5369070aaa007c1f8eb66bed7c731b4beb8915961e636d832 rebuilt index: <not found> committed index: (37, 11635176)
ID: c02e446eef1a99b7109d5a43920b889510b4a65ebf4279b4942bc3ea39af5e3d rebuilt index: <not found> committed index: (37, 9610778)
ID: d8e8e5279d2f2caef26390895778474d226943d96f05134014e02c0a3714f19b rebuilt index: <not found> committed index: (37, 2864907)
ID: 819377d08dc31629177681e3f0094bf8411b02f91decc6e8bc4df0698b31e43d rebuilt index: <not found> committed index: (37, 25987415)
ID: 2c9f92db625ca9e5545d2732e92c3335ad3a7e982e45d166a99d83196a5d30a8 rebuilt index: <not found> committed index: (37, 16685340)
ID: 4dfc2f373d89104d6f07683855081c28de0511b89401cd08c2a764671dc1daf9 rebuilt index: <not found> committed index: (37, 9708258)
ID: a242417d95e9975affd8bfca3f0ce09b3ea033bd62d77fa63d35641b22addefc rebuilt index: <not found> committed index: (37, 16423135)
ID: 421072498497083e8118ecf13e7972324b1400d61c6ab9e53eae9ea0dc76e828 rebuilt index: <not found> committed index: (37, 9750260)
ID: fc62ca9848d8dd0c7740032bec85747affb9bbb20f896b230808fc2c23d12036 rebuilt index: <not found> committed index: (37, 13670158)
ID: 62a27ad6c110b9af1400cc2567e36804a42b65151e5de5d2f796d6b42947a810 rebuilt index: <not found> committed index: (37, 17038796)
ID: befba82e30d23d55d73e44e5ee5e55d9e3c8a7a989d2af4c492d56514d925051 rebuilt index: <not found> committed index: (37, 13821679)
ID: c59e61d011c0a7a878f6d52830a07c33b63f9e3da7449d408a346ce931c83a6b rebuilt index: <not found> committed index: (37, 17031073)
ID: 47fcff2cde9e9bb107b8d3e7101a3bdb948d9500a339233ce34070e5b79fa18f rebuilt index: <not found> committed index: (37, 9706691)
ID: 061e154af0fde36ab5dea929a108e0d332bb029e99a5329dc63b9edd38700386 rebuilt index: <not found> committed index: (37, 4429663)
ID: 981c33417d1b734e66f8f825eff6f3927f4d92154a4780592b1b11b911a5bd20 rebuilt index: <not found> committed index: (37, 14188677)
ID: 66a5bcc9dc64a9741b92c613033d3867e86733274cca10b76d3aaa275cf1756b rebuilt index: <not found> committed index: (37, 16685177)
ID: ff5c2f803efb80ca1e003b1c5bd8942ecfe3f0d30ad79d6c7fe865bb6dbf26e1 rebuilt index: <not found> committed index: (37, 6153736)
ID: a973d396eefebd6ebebacf65b7bbcb6a40c3bfbc55d11faed1750fe3199922da rebuilt index: <not found> committed index: (37, 26076612)
ID: df268b49ed5a44cea7d02ec3dc58056c17bfaa15160dca7f18302388c3b92148 rebuilt index: <not found> committed index: (37, 9750424)
ID: 840e0e315c7bf94a23b2d476b62804f09f9cb7ea292c0722edbef5ec5080cf49 rebuilt index: <not found> committed index: (37, 6611035)
ID: 6c18ba3859c81a110927861235ccd9fd4f07bbc9c9666b6e21178fe9c2addf7b rebuilt index: <not found> committed index: (37, 2495599)
ID: e536f355c3f4b65e9c4ce6507617f6782e41ad840ebfecf6967bb97109f558b8 rebuilt index: <not found> committed index: (37, 13516566)
ID: 4fecf00ad447b88fe2e31329e0a037bde961168aece7817de77dae412ec58ddb rebuilt index: <not found> committed index: (37, 9441984)
ID: 0e5a3774e6ba7ad02b862d35b94b09d712e0297b3758825fff2ea13446d0e93e rebuilt index: <not found> committed index: (37, 9741616)
ID: 9e2a7044eb5f1f0d7b70014cc12af5459e67436eeff9f249cd42bc002034e4d7 rebuilt index: <not found> committed index: (37, 16936116)
ID: 3fc2b5dad03bbf0287e3fb09a2d6a184f05072a28315fb4fd92ab67c95bfd966 rebuilt index: <not found> committed index: (37, 16867101)
ID: 00faf8115d5dcea847992eb80bec88188d90ca3d1cd6c7069f1454e3802fe51c rebuilt index: <not found> committed index: (37, 1744927)
ID: a10fe725be9fa2a6b3661a10ddcd61f79ac52ee3340620ff23f3d77316b43930 rebuilt index: <not found> committed index: (37, 16960738)
ID: c5fc9011276980b7e701226fe27d08fefbe9b5a1dafc38f5c0002cf77605904c rebuilt index: <not found> committed index: (37, 16961931)
ID: 9ad10fe6901d27fcd8dc704f056a465777522cf3e079148a6dfc1580bbc8fd9c rebuilt index: <not found> committed index: (37, 17084281)
ID: babc3dd0acd7369d940e236fd662f3d08fb1a37853a7138e4c9500b494f69641 rebuilt index: <not found> committed index: (37, 17010668)
ID: e4207a31803d60a5524e7860927810a77baacf40a99b6dc73f5104f061077282 rebuilt index: <not found> committed index: (37, 15103990)
ID: 519425a181839f30e6c7fa7e60b9a8b5f14da5ef4d1397c4d53adbacaa6ff19a rebuilt index: <not found> committed index: (37, 16958159)
ID: 9c80ac8cbd8e38d2b89dd6176d199f78fbd0971d554ddf6fb8fd5efdd94c3c6a rebuilt index: <not found> committed index: (37, 9441724)
ID: a65d8869d98e8909aac22c948a25804f1d1eb8c458a20358f2ebad7a34b7525d rebuilt index: <not found> committed index: (37, 25178260)
ID: 5b463c513d72beca29e3a10c2899baf91fc9a784db3637f311007a2cc064390e rebuilt index: <not found> committed index: (37, 17068416)
ID: ae661f6f8a6450003d9ea504eb59680c4a7fc630a3bb3cfe958314cf3df7edc4 rebuilt index: <not found> committed index: (37, 16308860)
ID: d127f92ff8636583a7de25dfa12a992e80ed6931eb3ecdd34ec7a7ee9cd23fd0 rebuilt index: <not found> committed index: (37, 16960300)
ID: 8072c678019444d1f8a410489343d595c4fb35e984556b482e743d88e2edf167 rebuilt index: <not found> committed index: (37, 9494845)
ID: 32e6c2eaf4996cb54150f562a2e0c05db787a8dee7bc66758a20f0072a4c1428 rebuilt index: <not found> committed index: (37, 12606828)
ID: 497f2b823fa7c553723477c23c10afbd48e1d6960bb846f8fcbd874572d52001 rebuilt index: <not found> committed index: (37, 17177608)
ID: e6353038c8161ed83e0d35789f5b1ef9de9d0428ea4c56b3039999ab332eef41 rebuilt index: <not found> committed index: (37, 16432099)
ID: 59b8b6b8b0889a80f71cee248021b79aaa59ef2d3828c05c565a6dfbf5ec7ac4 rebuilt index: <not found> committed index: (37, 17079852)
ID: 7cd696d6e4f6f5b0fc903ac4e994ee797345165619ea87ff5dacc267cdd7de38 rebuilt index: <not found> committed index: (37, 7709812)
Finished full repository check, errors found.
Some errors were found while checking the backup integrity!
Check the AIO interface for advices on how to proceed now!
And repair logs:
Checking the backup integrity and repairing it...
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!
Type 'YES' if you understand this and want to continue: Starting repository check
Data integrity error: Invalid segment magic [segment 37, offset 0]
attempting to recover /mnt/borgbackup/borg/data/0/37
Local Exception
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1040, in check
objects = list(self.io.iter_objects(segment))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1545, in iter_objects
raise IntegrityError(f'Invalid segment magic [segment {segment}, offset {0}]')
borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment magic [segment 37, offset 0]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/borg/platform/base.py", line 243, in __exit__
os.replace(self.tmp_fname, self.path) # POSIX: atomic rename
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/borgbackup/borg/data/0/37-hseny2eo.tmp' -> '/mnt/borgbackup/borg/data/0/37'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main
exit_code = archiver.run(args)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run
return set_ec(func(args))
^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
return method(self, args, repository=repository, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 343, in do_check
if not repository.check(repair=args.repair, save_space=args.save_space, max_duration=args.max_duration):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1045, in check
self.io.recover_segment(segment, filename)
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1576, in recover_segment
with SaveFile(filename, binary=True) as dst_fd:
File "/usr/lib/python3.11/site-packages/borg/platform/base.py", line 245, in __exit__
safe_unlink(self.tmp_fname) # rename has failed, clean up tmp file
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/helpers/fs.py", line 221, in safe_unlink
os.unlink(path)
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/borgbackup/borg/data/0/37-hseny2eo.tmp'
Platform: Linux b0ee91006d48 5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64
Linux: Unknown Linux
Borg: 1.2.6 Python: CPython 3.11.5 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 28 CWD: /
sys.argv: ['/usr/bin/borg', 'check', '-v', '--repair', '/mnt/borgbackup/borg']
SSH_ORIGINAL_COMMAND: None
Some errors were found while checking and repairing the backup integrity!
borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment entry header [segment 37, offset 27231398]: unpack requires a buffer of 41 bytes
Looks like the segment file 37 in that repo is corrupted.
Usually borg check --repair should be able to deal with that, but looks like not in that case.
So, I'ld suggest to list the files below repo_dir/data/0/ and check if there is something special with the file 37 in there (maybe zero size?). If the file is less that 17 bytes, you can just delete it and run borg check --repair again.
Another question is of course why you have damaged files in your repo, that could point to a hw or sw problem below borg.
Thank you for a quick response, will check segment file 37 right away and give back results.
Interestingly enough I actually did full-upgrade yesterday and noticed kernel upgrade during that. Could that be the reason for incompatibility? Quite low chance I would assume, taking into account that Raspberry Pi is used only as a mean to mount the hard drive, but I have no idea as to what could have caused that
Actually as I see there are 4 different files with name starting from 37:
╰─➤ ls -lah 37*
-rwxrwxrwx 1 root root 3,5M wrz 18 11:27 37
-rwxrwxrwx 1 root root 8 wrz 29 07:02 37-0dyqb8o9.tmp
-rwxrwxrwx 1 root root 8 wrz 29 07:16 37-hseny2eo.tmp
-rwxrwxrwx 1 root root 8 wrz 29 06:16 37-i6x_cqlp.tmp
Does this indicate that something happened during the backup?
You can ignore / delete the *.tmp files.
But the file 37 looks like it has some content, maybe try to read it / look inside .
Also, a fsck on that filesystem might be a good idea (if you think your hw is ok, otherwise first fix the hw).
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/borgbackup/borg/data/0/37-hseny2eo.tmp'
Interesting that you have that file in your ls output.
Label "hw-issue" (or kernel / fs issue) - cause of corruption
Label "bug" because borg check --repair crashed.
37 is a file, so unfortunately I can not ls into that. What can be done to be able to create backups again? Is deletion of 37-like files still a valid step?
Still when I am thinking about that - there should be no way that upgrade affected because setup is like that:
- Cloud VPS with nextcloud-aio, which uses borg docker for backup and directory is an NFS-mounted folder
- Raspberry Pi 4, which has hard drive connected with USB3.0, which is NFS-mounted in VPS
I would understand if problem would appear on a filesystem on SD card, but hard drive - rather strange.
I analysed a somehow similar issue there: https://github.com/borgbackup/borg/issues/7221#issuecomment-1363193614
That "unpack requires a buffer of 41 bytes" points to a "short or empty read" where borg expected to be able to read a segment entry header (which is 41b long).
Can you read the file 37 (like e.g. copy it to /tmp)? Does it start with the magic string "BORG_SEG"?
That sounds like that magic is not at the beginning of the file:
borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment magic [segment 37, offset 0]
cat 37 | grep BORG is empty.
File itself appears like it contains some data:
╰─➤ head -n 2 37 1 ↵
���� LExifMM� ���(1�2�< �BC�i��HAppleiPhone XHH16.62023:09:17 13:47:23iPhone X#�������"�'�0232���Ȑܐ���
����
Those 37-*.tmp files do contain BORG_SEG text only though
That looks like a photo from an iphone, not like a borg segment file.
Make sure hw and kernel is ok (maybe use the old kernel, before upgrade), then run fsck on that filesystem.
After that, try borg check --repair again.
I got a bit scared by fsck's message:
WARNING!!! The filesystem is mounted. If you continue you ***WILL***
cause ***SEVERE*** filesystem damage.
Also unfortunately hard drive is used by other systems, so would not be easy / fast to turn those off to unmount hard drive.
On the other hand running repair again yielded with the same results - it failed to find a file which is present in that directory:
Checking the backup integrity and repairing it...
This is a potentially dangerous function.
check --repair might lead to data loss (for kinds of corruption it is not
capable of dealing with). BE VERY CAREFUL!
Type 'YES' if you understand this and want to continue: Starting repository check
Data integrity error: Invalid segment magic [segment 37, offset 0]
attempting to recover /mnt/borgbackup/borg/data/0/37
Local Exception
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1040, in check
objects = list(self.io.iter_objects(segment))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1545, in iter_objects
raise IntegrityError(f'Invalid segment magic [segment {segment}, offset {0}]')
borg.helpers.errors.IntegrityError: Data integrity error: Invalid segment magic [segment 37, offset 0]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/borg/platform/base.py", line 243, in __exit__
os.replace(self.tmp_fname, self.path) # POSIX: atomic rename
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/borgbackup/borg/data/0/37-fm4dod5p.tmp' -> '/mnt/borgbackup/borg/data/0/37'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5324, in main
exit_code = archiver.run(args)
^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 5255, in run
return set_ec(func(args))
^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 183, in wrapper
return method(self, args, repository=repository, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/archiver.py", line 343, in do_check
if not repository.check(repair=args.repair, save_space=args.save_space, max_duration=args.max_duration):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1045, in check
self.io.recover_segment(segment, filename)
File "/usr/lib/python3.11/site-packages/borg/repository.py", line 1576, in recover_segment
with SaveFile(filename, binary=True) as dst_fd:
File "/usr/lib/python3.11/site-packages/borg/platform/base.py", line 245, in __exit__
safe_unlink(self.tmp_fname) # rename has failed, clean up tmp file
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib/python3.11/site-packages/borg/helpers/fs.py", line 221, in safe_unlink
os.unlink(path)
FileNotFoundError: [Errno 2] No such file or directory: '/mnt/borgbackup/borg/data/0/37-fm4dod5p.tmp'
Platform: Linux 4e91091ea5a3 5.15.0-84-generic #93-Ubuntu SMP Tue Sep 5 17:16:10 UTC 2023 x86_64
Linux: Unknown Linux
Borg: 1.2.6 Python: CPython 3.11.5 msgpack: 1.0.5 fuse: llfuse 1.4.2 [pyfuse3,llfuse]
PID: 28 CWD: /
sys.argv: ['/usr/bin/borg', 'check', '-v', '--repair', '/mnt/borgbackup/borg']
SSH_ORIGINAL_COMMAND: None
Some errors were found while checking and repairing the backup integrity!
File is present:
╰─➤ ls * | grep fm4dod5p 2 ↵
37-fm4dod5p.tmp
Name of the file causing error is different this time though
Hm, how can I solve the issue without unmounting the hard drive? I do not mind losing that particular backup, I am positive that hustle with stopping services to unmount safely is not worth it. Maybe if the issue would appear again I would go through fsck
Honestly I am even thinking about completely removing the backup folder, as that is my private data not used anywhere else, so no hard feelings losing the data here.
Due to deduplication, a segment file usually contains chunks referenced by multiple backup archives.
So, if you lose a segment file, it might affect multiple archives.
Also, not checking/fixing (fsck) a filesystem that might have fs-level corruption is a bad idea.
That iphone photo file data getting referenced from the borg segment file is a clear indication of fs-level confusion / corruption.
I see what happened with 37 being a file with data while 37-....tmp being present - I did not turn off backuping in nextcloud-aio as well as ran backup manually once or twice. This night it also failed to backup data and new 37-....tmp files appeared, having BORG_SEG inside
The tmp files come from borg trying to repair that segment file. Usually that repair should succeed (but here, it does not - likely because of the corrupted filesystem) and usually the tmp file should be removed in any case (but here it isn't).
Decided to change backup directory for now. I will unmount disk and will run fsck when I have a little bit more free time (end of next week if nothing else will get into my way)
Thank you greatly for help and explanation! :heart: