restic
Restic damaged pack files
Hello everyone,
I ran the command restic check --read-data -r /DATA/backup_local and encountered several errors. Here’s the output:
using temporary cache in /DATA/tmp/restic-check-cache-3188513859
create exclusive lock for repository
enter password for repository:
repository 8eb1a62a opened (version 2, compression level auto)
created new cache in /DATA/tmp/restic-check-cache-3188513859
load indexes
[0:02] 100.00% 53 / 53 index files loaded
--------------------------------------------------The next 40960 lines are like this---------------------------------------------------------------------------------
pack ce31fc547300f27c50237e3a2e0c3921162e4d0e86245094a8703cc03230f33f contained in several indexes: {4be1f0fd 8a57cd6b}
...
...
...
Duplicate packs are non-critical, you can run `restic repair index' to correct this.
check all packs
check snapshots, trees and blobs
[1:23] 100.00% 3901 / 3901 snapshots
read all data
check successful on second attempt, original error pack 30f9248f2b9054d776d7248dc311eaf62fbfa208c860391b6df3db82fb4875e9 contains 1 errors: [blob 394644208d47b374f9284eeaf8c66e9edc014bcc390e9c2b050668b3570b7d97: read blob <data/39464420> from 30f9248f: wrong data returned, hash is fd48abc29dfd5cc05102cd21381d3811ca7672d0286ac24fa4041a8ac38e377f]
check successful on second attempt, original error pack 000001bcaa440ee7559dbf0dddf1d795686fff24e0558d389ad963fd42a53485 contains 1 errors: [blob 0763ea09c0d9dbef590695e54e20c7463f619c26b77c657443b02dbeaa248aaf: read blob <data/0763ea09> from 000001bc: wrong data returned, hash is 7f1c8508bd5fd592b1d52e8a90205de694d0cbe09ccc04471eaec54aa56a38e9]
check successful on second attempt, original error pack d014b3922679a5553d891ca52782d2e823fac36647706edcf528dd00788652ce contains 1 errors: [blob a6215d1b07a635f44fc9eb5329301904c51716c22d09a2b95a588acbc165840e: read blob <data/a6215d1b> from d014b392: wrong data returned, hash is 900a69ef8783d2496adf87a13641ff0f521c578b4ae2f8798b4af11961ccf0ab]
check successful on second attempt, original error pack a032c0a9300a090c8812ab953fbc741d71ae17255fe9c5a088da0fd191cf25e1 contains 1 errors: [blob 6a22bf4a778eab465486aa77d2383545df024ef7841f4f0d61983438f124bdcd: decrypting blob <data/6a22bf4a> from a032c0a9 failed: ciphertext verification failed]
check successful on second attempt, original error pack 313cd82fcfcf1a120f09dd41218698b166e980d8486b28431041c8096d2b9494 contains 1 errors: [blob 2609a60a6393b403c2a18c85d49ad70f5f855527032d184be346f4c28c35ebc3: read blob <data/2609a60a> from 313cd82f: wrong data returned, hash is 0b17d242d2a894f31ec930c07091d01d9fa65ce8ce7448d7bd7dc59d754b3f51]
pack 21d3e3203ab1dd9c987f2ce7a327422d567fd24f2e4a9311483ce9c34fdd92d6 contains 1 errors: [blob 637f7c61402503596ced050e78621f6759fb27dd5f680af74f84ca16ecf5da82: read blob <data/637f7c61> from 21d3e320: wrong data returned, hash is 9f959b902ebea57844028d840d0dc49114d46c5084f9b57b735b3c2d31960d49]
pack b1565f1a90a006fe1495afe1b27a30a65d4baa4da72e122458099e6d756f92ff contains 2 errors: [blob 5a0515090e8390b8e4e96707fef75c6eceadd9e68f29f06141126a925c23ff39: decrypting blob <data/5a051509> from b1565f1a failed: ciphertext verification failed unexpected pack id 06ca7fd727c0b33e33c08d5fa544f4a37ce2896032b60d63e73a53fcb641386f]
pack 9376a0a813f80e36cb8a05aa57a33bd98679f99cb062ea5476ff2f6c547bb799 contains 1 errors: [blob 0ff842f93e8e4bd40942dcf8a6c3015f982e7c5b625d0f38ef6bcd7d9c1ba58a: read blob <data/0ff842f9> from 9376a0a8: wrong data returned, hash is 7c74e4908e66390e4200f3d6ae6ac9a35b1c5b9c4bc56280ae3cede55e9882cd]
pack 27cacfcf55eea8760e05d6af141b9811c6fc7ea8c1275babfdd2ce53de7c6475 contains 1 errors: [blob 3b7f33b1aca3af19fbcb3e69f800af78ee6e0c7e10eb51a4aa616ce430d67a1f: decrypting blob <data/3b7f33b1> from 27cacfcf failed: ciphertext verification failed]
pack 0d8797226cd4bb7a7db9aed62149994152629ffd07520fdfeffae239adab4c23 contains 2 errors: [blob 52a20fd15f77dabae0b37f5fbcf8e1657adb0add0899f1f4ea04f1035452c8a2: decrypting blob <data/52a20fd1> from 0d879722 failed: ciphertext verification failed unexpected pack id 54150a66461bc31151497ee2dd0256bf16b4315d0ff321dec715ed4c8323e845]
pack 4d4e728791ca08006b075bd9397c35e986befae095aeed61b17924022a3b0973 contains 1 errors: [blob ab6fb6962eb0c7a1ba493b20d45658c05b2ba83550a4d8024bfd0270bf0f5e30: decrypting blob <data/ab6fb696> from 4d4e7287 failed: ciphertext verification failed]
[3:27:39] 100.00% 73362 / 73362 packs
The repository contains damaged pack files. These damaged files must be removed to repair the repository. This can be done using the following commands. Please read the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html first.
restic repair packs b1565f1a90a006fe1495afe1b27a30a65d4baa4da72e122458099e6d756f92ff 9376a0a813f80e36cb8a05aa57a33bd98679f99cb062ea5476ff2f6c547bb799 27cacfcf55eea8760e05d6af141b9811c6fc7ea8c1275babfdd2ce53de7c6475 0d8797226cd4bb7a7db9aed62149994152629ffd07520fdfeffae239adab4c23 4d4e728791ca08006b075bd9397c35e986befae095aeed61b17924022a3b0973 21d3e3203ab1dd9c987f2ce7a327422d567fd24f2e4a9311483ce9c34fdd92d6
restic repair snapshots --forget
Damaged pack files can be caused by backend problems, hardware problems or bugs in restic. Please open an issue at https://github.com/restic/restic/issues/new/choose for further troubleshooting!
Fatal: repository contains errors
Given the variety of errors, I would appreciate any advice on how to proceed. For context, the database size is 1.2 TB, and there are 154,481 files in the /DATA/backup_local directory. Thank you!
You forgot to mention which version of restic you are using (restic version), how the /DATA/backup_local directory is mounted and what filesystem it is.
This could very well be a hardware issue on the client or perhaps on the storage level, or an issue with whatever filesystem you have that that on. Did you run the suggested commands?
@rawtaz Thanks for the reminder. Here’s the data:
restic 0.17.3 compiled with go1.23.3 on linux/amd64
lsblk -f
NAME FSTYPE FSVER LABEL UUID FSAVAIL FSUSE% MOUNTPOINTS
sda
├─sda1 vfat FAT32 E369-F00C 505.1M 1% /boot/efi
├─sda2 ext2 1.0 4477f9e9-3a41-1234-bf85-ad2876a3c32a 307M 27% /boot
└─sda3 LVM2_member LVM2 001 HJ3tZe-6xqz-vnPJ-pw10-OHaD-din7-e2DkWE
├─lego--vg-root ext4 1.0 fba12a6e-9a00-4f01-8654-c8675c9fe737 16G 16% /
├─lego--vg-var ext4 1.0 feb7e02f-8eaa-475a-9c8a-6a2f92c9700b 5.3G 19% /var
├─lego--vg-swap_1 swap 1 e045355f-b26e-479b-b47e-25eaef086cf5 [SWAP]
├─lego--vg-tmp ext4 1.0 8bff46a4-dcd1-4c18-8c8e-8fd08ccdd03b 1.2G 0% /tmp
└─lego--vg-home ext4 1.0 e61b6cc1-edc7-43e7-b918-5c8697c1369f 73.1G 2% /home
sdb
└─sdb1 linux_raid_member 1.2 lego:0 3a2363d1-1234-f147-51ba-6460a908dac1
└─md0 crypto_LUKS 2 614efd05-cde2-41c1-a5c0-3b8c2bbd5b54
└─DATA ext4 1.0 ac05ca7f-2b41-41ad-a426-d099535360ca 8.2T 45% /DATA
sdc
└─sdc1 linux_raid_member 1.2 lego:0 3a2363d1-1234-f147-51ba-6460a908dac1
└─md0 crypto_LUKS 2 614efd05-cde2-41c1-a5c0-3b8c2bbd5b54
└─DATA ext4 1.0 ac05ca7f-2b41-41ad-a426-d099535360ca 8.2T 45% /DATA
Two stress tests were carried out successfully, each lasting 10 minutes and conducted consecutively.
stress -c 13 -i 4 --verbose --timeout 10m
stress: dbug: [2282613] <-- worker 2282617 signalled normally
stress: dbug: [2282613] <-- worker 2282615 signalled normally
stress: dbug: [2282613] <-- worker 2282623 signalled normally
stress: dbug: [2282613] <-- worker 2282616 signalled normally
stress: dbug: [2282613] <-- worker 2282614 signalled normally
stress: dbug: [2282613] <-- worker 2282620 signalled normally
stress: dbug: [2282613] <-- worker 2282624 signalled normally
stress: dbug: [2282613] <-- worker 2282626 signalled normally
stress: dbug: [2282613] <-- worker 2282618 signalled normally
stress: dbug: [2282613] <-- worker 2282622 signalled normally
stress: dbug: [2282613] <-- worker 2282625 signalled normally
stress: dbug: [2282613] <-- worker 2282627 signalled normally
stress: dbug: [2282613] <-- worker 2282628 signalled normally
stress: dbug: [2282613] <-- worker 2282629 signalled normally
stress: dbug: [2282613] <-- worker 2282630 signalled normally
stress: dbug: [2282613] <-- worker 2282621 signalled normally
stress: dbug: [2282613] <-- worker 2282619 signalled normally
stress: info: [2282613] successful run completed in 600s
@IV-R I appreciate the suggestions, but the steps you've described are beyond my programming skills, and I'm afraid I won't be able to handle them.
@lego-2024 The check successful on second attempt, original error pack error along with lots of different data corruptions types is very likely caused by data corruption in the CPU or RAM of the host running restic. Restic 0.17.3 has been available for some time and we're not aware of any data corruption issues in restic itself. Please run prime95 / memtest to make sure your hardware is ok.
The pack ce31fc547300f27c50237e3a2e0c3921162e4d0e86245094a8703cc03230f33f contained in several indexes log messages are just a warning and not particularly relevant here.
check successful on second attempt, original error pack a032c0a9300a090c8812ab953fbc741d71ae17255fe9c5a088da0fd191cf25e1 contains 1 errors: [blob 6a22bf4a778eab465486aa77d2383545df024ef7841f4f0d61983438f124bdcd: decrypting blob <data/6a22bf4a> from a032c0a9 failed: ciphertext verification failed] can only be explained by data corruption on the host running restic. This is NOT a problem with the storage / filesystem.
@IV-R I appreciate the suggestions, but the steps you've described are beyond my programming skills, and I'm afraid I won't be able to handle them.
Those suggestions look like AI-generated garbage that isn't any useful for debugging the issue you're facing.
Alright, I'll begin the system testing with Prime95 and will share the results once they're available. Thanks, everyone!
After running the Prime95 (version 30.19) for 24 hours, the system has completed the torture test without errors/warnings, suggesting that it is reliable and functioning as expected.
Check of the filesystem is also OK sudo fsck.ext4 -n /dev/mapper/DATA e2fsck 1.47.0 (5-Feb-2023) /dev/mapper/DATA: clean, 709444/274659328 files, 1967725328/4394544384 blocks
Here are additional details: System RAM: 8GB Size of the virtual mounted backup: 1.2 PB Number of files in the virtual mounted backup: almost 150,000,000 Total files backed up: approximately 130,000
After running the Prime95 (version 30.19) for 24 hours, the system has completed the torture test without errors/warnings, suggesting that it is reliable and functioning as expected.
And yet googling for "check successful on second attempt" only yields two results. This issue and one thread in the restic forum, but that thread is related to download issues (not the case here). So there must be something about your system that caused/causes those errors. As most errors do not include "unexpected pack id", this only leaves CPU and memory as the potential culprits. There's one rather odd pattern in the initial error log: the first few packs could be verified on a second attempt, but that stopped after the first few files.
prime95 has been rather useful to detect issues, but it doesn't test all features of a modern processor. The errors above could also be triggered by a problem with the crypto hardware acceleration, but I have no clue how to diagnose that.
Do the errors reappear if you run check again?
Two executions produced identical results, with the only variation between them being the order of the data. For example, in the first execution, the output was:
.......
restic repair packs 21d3e3203ab1dd9c987f2ce7a327422d567fd24f2e4a9311483ce9c34fdd92d6
9376a0a813f80e36cb8a05aa57a33bd98679f99cb062ea5476ff2f6c547bb799
27cacfcf55eea8760e05d6af141b9811c6fc7ea8c1275babfdd2ce53de7c6475
0d8797226cd4bb7a7db9aed62149994152629ffd07520fdfeffae239adab4c23
4d4e728791ca08006b075bd9397c35e986befae095aeed61b17924022a3b0973
.......
On the other hand, the output from the second execution was:
.......
restic repair packs 27cacfcf55eea8760e05d6af141b9811c6fc7ea8c1275babfdd2ce53de7c6475
0d8797226cd4bb7a7db9aed62149994152629ffd07520fdfeffae239adab4c23
4d4e728791ca08006b075bd9397c35e986befae095aeed61b17924022a3b0973
21d3e3203ab1dd9c987f2ce7a327422d567fd24f2e4a9311483ce9c34fdd92d6
9376a0a813f80e36cb8a05aa57a33bd98679f99cb062ea5476ff2f6c547bb799
.......
This pattern holds true for the entire output as well.
The distinction between the original run and the second and third executions is that the message "check successful on second attempt" was not present in the latter two, and the pack b1565f1a90a006fe1495afe1b27a30a65d4baa4da72e122458099e6d756f92ff is now error-free.
P.S.: The first run lasted 3 hours and 30 minutes, while the second run took 8 hours and 12 minutes.
The distinction between the original run and the second and third executions is that the message "check successful on second attempt" was not present in the latter two
Hmm, having check consistently report the same pack files essentially means that those pack files are corrupted on disk. But it still makes zero sense to me where the "check successful on second attempt" came from. Do you have a second system you could use to run check --read-data to make sure that there's really nothing wrong with the host?
Well, this is going to be a VERY long story. The only other system I have with such large storage is a Pi3 with an external disk. I'll give it a try and report back in a few days.
Hello! After 90 hours of testing, the new system showed the following result:
40959 entries "pack ...xxxx... contained in several indexes: {...xxx... ...xxx...}" This is non-critical, you can run `restic rebuild-index' to correct this check all packs check snapshots, trees and blobs [16:59] 100.00% 3901 / 3901 snapshots read all data Pack ID does not match, want 2003efd822282b126b4329994b37c9e57d4d0b07f1aeef9f39bbef668cd36c9d, got 8d247d5ec61a51eace23e4ef02b797a5dcbf6adb5f4be4787b28ffc06deed007 Pack ID does not match, want 20f4ce1250c1ae831da4cd1af16c242bb20328d9cf13f4aaf49c90f02ea47e1d, got 968b54d7fbddd315890168b2cccb53622e4cfd39ede7c9d692a94ec156357499 Pack ID does not match, want 21e03bbd442919b948f37bb3e5cafc3ca0b03df4d220e5751fd88410fbebe3e2, got 9ff9db3b4eeed3b8ba25df0c2cfcfb388b71fe5cf352e5f44c77e93c725b6c19 Pack ID does not match, want 2160bd0c295548ed605748ba767d6deffa3cb169c3b7eb18eed040d15ca312a4, got 449ab278ece945dfd3fcec3692938da4d40f3d40e0a554e9c60f26f241714640 Pack ID does not match, want 21743b189ec1c4184d34f97a467c511132a617355c41c75a1467166c89e762b1, got 6a4bcac177f0250993f533c866296316918f4e526a7d83f9baa3b3485d0718f9 pack 21d3e3203ab1dd9c987f2ce7a327422d567fd24f2e4a9311483ce9c34fdd92d6 contains 1 errors: [blob 637f7c61402503596ced050e78621f6759fb27dd5f680af74f84ca16ecf5da82: read blob <data/637f7c61> from 21d3e320: wrong data returned, hash is 9f959b902ebea57844028d840d0dc49114d46c5084f9b57b735b3c2d31960d49] Pack ID does not match, want 21e4c05aa0f4de08ea8b90674d9547f11b0b15b668f77797616d111ad343dc36, got 1a13503e7d9de301a31ba1e55ffecc58c4b79d41b18e155f15e824a726cef231 Pack ID does not match, want 214e4d587013b2e1e07db8d8ed2181628e8ba69331dbf002c0639c0682d9089f, got af2adc014fb129a97059025764520948aea555361fa482b7e542269aa5efe550 Pack ID does not match, want 22c3a3d1484c1ab485d24c6f7d69b04a9fc66a69f3b6e7081ebebbb29e78803d, got b571a9154539f6be9e98fb66dc28006aaf7993fdd6a103b9a551c1a5a6c6ce43 Pack ID does not match, want 220104a4fcca91b8b3070e1a17a29823a15709a14389fde95200fb1f691c8c81, got c0e9eab5370e12d09be7172101ecb940946937b3f472330846ee6804449572f9 Pack ID does not match, want 22a171304c63e6d5a40ac7e7ff05f49452216dcb0ea1a2f92a69967afbc0a49e, got d1cb1e79059ff478d8a9d65eecb7874faba7c644e88ad63498833d4d3916d4a8 Pack ID does not match, want 23590ec13ae302ced095cfd8746f9effe72683bd47baa40f52746968e7f4ec6f, got 1c222eecd3a32f0f08876c921a4d2dcd15d1be920448cc4973cbe49c65882057 Pack ID does not match, want 231da6d5af4559f55ccd5de4b136ac23adc305e75901394584efd3d810fbfb4f, got a442387c58bfc14201a0beae4c7cd839f8f0eca05385252f6646d5bd68033317 pack 9376a0a813f80e36cb8a05aa57a33bd98679f99cb062ea5476ff2f6c547bb799 contains 1 errors: [blob 0ff842f93e8e4bd40942dcf8a6c3015f982e7c5b625d0f38ef6bcd7d9c1ba58a: read blob <data/0ff842f9> from 9376a0a8: wrong data returned, hash is 7c74e4908e66390e4200f3d6ae6ac9a35b1c5b9c4bc56280ae3cede55e9882cd] Pack ID does not match, want 248e04c2f1700ca5994e32fba86826982a38f0b1587ebf8f0b92897b9ced38e3, got d0e88de3d0f8eeb6f84814f202566a488bef22c9446ca3132647364983da04a2 Pack ID does not match, want 24a7af1e2da7e87cedb31bb07addd105f81bb02c64695a604bf789695e20314a, got 35e030b9c2989f1975ecbcae518fa955cf805cad529cb6ea84d840a1f93e7363 Pack ID does not match, want 24b1cabcd32fa030aed8690733f2330e356409da0d66b654abcf76ba37b18760, got b6d0f2d3f334953fe059f4e6a04a7bc259c58da5f70169167210b1d9cedcbb3d pack 27cacfcf55eea8760e05d6af141b9811c6fc7ea8c1275babfdd2ce53de7c6475 contains 1 errors: [blob 3b7f33b1aca3af19fbcb3e69f800af78ee6e0c7e10eb51a4aa616ce430d67a1f: ciphertext verification failed] Pack ID does not match, want 2a35895a357907df43fa119374de2803d8bda6ae27d59b2dd04c865e717b5280, got 3786a27b25d57d6eb99e0e5ea69cf1d53cb6dbb91100434a628841598630be2c Pack ID does not match, want 0b19b8023384303970c6043c3dcdcbeca3dd08e49e1cac9241adb0dbb0769af2, got d8ba56be547f093325708ad4bc7e135191f0ba8596240aef14cfb0eaeccc7b4e Pack ID does not match, want 1c1864942dc533e6560db9e554d84398f4a3e414b191f11b79bd58c376e78244, got 195e6cfab82758965640d939177ef078275fc55b03bdfec328364e445e2f000e pack 4d4e728791ca08006b075bd9397c35e986befae095aeed61b17924022a3b0973 contains 1 errors: [blob ab6fb6962eb0c7a1ba493b20d45658c05b2ba83550a4d8024bfd0270bf0f5e30: ciphertext verification failed] Pack ID does not match, want 0d8797226cd4bb7a7db9aed62149994152629ffd07520fdfeffae239adab4c23, got 54150a66461bc31151497ee2dd0256bf16b4315d0ff321dec715ed4c8323e845 Pack ID does not match, want 5ed8eea8226bc62425a8273cb45ef026fd1728ee6288a6f0787edfea8c2ecb13, got 76f55cc32b40cabc5c2cc7a3f162ac1358a57d10985da5c101fa33c117b6f07d Pack ID does not match, want 1f1eaaf3e42737d32f568da5de584a6e59964aeaaf6beb89d14bb3c0183987a5, got d39b370696f9fb51964974226e1f57bc851abf987dce458868f04337410661f2 [89:14:06] 100.00% 73362 / 73362 packs Fatal: repository contains errors
The results are somewhat weird: the pack [...] contains 1 error are largely consistent with those from the initial list of errors. This means that they definitely got corrupted during the backup. Repairing those packs should work with the following command:
restic repair packs 9376a0a813f80e36cb8a05aa57a33bd98679f99cb062ea5476ff2f6c547bb799 27cacfcf55eea8760e05d6af141b9811c6fc7ea8c1275babfdd2ce53de7c6475 4d4e728791ca08006b075bd9397c35e986befae095aeed61b17924022a3b0973 21d3e3203ab1dd9c987f2ce7a327422d567fd24f2e4a9311483ce9c34fdd92d6 0d8797226cd4bb7a7db9aed62149994152629ffd07520fdfeffae239adab4c23
restic repair snapshots --forget
Pack ID does not match, want 2003efd822282b126b4329994b37c9e57d4d0b07f1aeef9f39bbef668cd36c9d, got 8d247d5ec61a51eace23e4ef02b797a5dcbf6adb5f4be4787b28ffc06deed007
Those errors are kinda worrying as they did not show up in initial check --read-data run and judging from the number of snapshots/packs the repository wasn't modified since.
Please run shasum -a256 data/20/2003efd822282b126b4329994b37c9e57d4d0b07f1aeef9f39bbef668cd36c9d data/20/20f4ce1250c1ae831da4cd1af16c242bb20328d9cf13f4aaf49c90f02ea47e1d data/21/21e03bbd442919b948f37bb3e5cafc3ca0b03df4d220e5751fd88410fbebe3e2 data/21/2160bd0c295548ed605748ba767d6deffa3cb169c3b7eb18eed040d15ca312a4 data/21/21743b189ec1c4184d34f97a467c511132a617355c41c75a1467166c89e762b1 data/21/21e4c05aa0f4de08ea8b90674d9547f11b0b15b668f77797616d111ad343dc36 data/21/214e4d587013b2e1e07db8d8ed2181628e8ba69331dbf002c0639c0682d9089f data/22/22c3a3d1484c1ab485d24c6f7d69b04a9fc66a69f3b6e7081ebebbb29e78803d data/22/220104a4fcca91b8b3070e1a17a29823a15709a14389fde95200fb1f691c8c81 data/22/22a171304c63e6d5a40ac7e7ff05f49452216dcb0ea1a2f92a69967afbc0a49e data/23/23590ec13ae302ced095cfd8746f9effe72683bd47baa40f52746968e7f4ec6f data/23/231da6d5af4559f55ccd5de4b136ac23adc305e75901394584efd3d810fbfb4f data/24/248e04c2f1700ca5994e32fba86826982a38f0b1587ebf8f0b92897b9ced38e3 data/24/24a7af1e2da7e87cedb31bb07addd105f81bb02c64695a604bf789695e20314a data/24/24b1cabcd32fa030aed8690733f2330e356409da0d66b654abcf76ba37b18760 data/2a/2a35895a357907df43fa119374de2803d8bda6ae27d59b2dd04c865e717b5280 data/0b/0b19b8023384303970c6043c3dcdcbeca3dd08e49e1cac9241adb0dbb0769af2 data/1c/1c1864942dc533e6560db9e554d84398f4a3e414b191f11b79bd58c376e78244 data/0d/0d8797226cd4bb7a7db9aed62149994152629ffd07520fdfeffae239adab4c23 data/5e/5ed8eea8226bc62425a8273cb45ef026fd1728ee6288a6f0787edfea8c2ecb13 data/1f/1f1eaaf3e42737d32f568da5de584a6e59964aeaaf6beb89d14bb3c0183987a5 and check whether the calculated hash matches the filename.
what does ls -la <list of data files from previous command return? Are those old or new pack files?
I’m feeling quite frustrated right now!
It looks like the main issue was caused by me. Upon examining the repository folder, I discovered another repository with the same name as the original. How and when this happened remains a mystery to me!
I want to take this chance to apologize to everyone who has contributed their efforts to help me.
P.S. Congratulations to the creators and everyone involved in the project.
It looks like the main issue was caused by me. Upon examining the repository folder, I discovered another repository with the same name as the original.
I don't really see how that could cause these specific issues (neither pack id mismatch, nor most of the other errors are compatible with that explanation. ciphertext verification failed could happen, but would look different than in the logs you've posted), but 🤷 . It there something left to debug here?
@lego-2024 Can you elaborate on what you found? You wrote "another repository with the same name as the original". What do you mean by that, are you saying that you had the second repository in the first one, or how could the second one that you found affect the first one that has been discussed here?
@rawtaz Yes, I discovered a repository with the same name (and the same password) within the original one, but I can't figure out how this happened! My original repository was created over a year ago, and during the first month, I made the backups manually. After that, I set up a crontab job. I suspect I accidentally created the second repository, but I can't determine how or when.
@MichaelEischer I'm also curious whether all the issues are due to the second repository. The only thing that stands out as abnormal is its existence.
Anyway, I will test the system with a manually created backup and will report back on the results of the check --read-data command
I conducted the following test:
1. Create a repository and add data with restic -r /path/to/restic_backup -v -p /path/to/pass_file backup /path/to/data_for_backup. Applied to seven different folders.
2. Check consistency and integrity with command restic check --read-data -r /path/to/restic_backup. The first check (as normal user) returned an error and the second check (as root) was successful..
3. Added the same seven folders to the repository (with some new data) this time as root.
Check again for consistency and integrity with command restic check --read-data -r /path/to/restic_backup. Again as root.
At the end of this process, it appears that something in my system is not working as expected. Here are the results from the check commands:
20:06:58user@backup:~$ restic check --read-data -r /path/to/restic_backup/ using temporary cache in /tmp/restic-check-cache-389693150 create exclusive lock for repository enter password for repository: repository 4336eb3a opened (version 2, compression level auto) created new cache in /tmp/restic-check-cache-389693150 load indexes [0:01] 100.00% 31 / 31 index files loaded check all packs check snapshots, trees and blobs [0:02] 100.00% 16 / 16 snapshots read all data check successful on second attempt, original error pack 4a6e87af7d3cc6d8be9c4a1c24123896a69031cdcd79374078745a267a8625fe contains 1 errors: [blob 4aaa51c660ed2d5c754539d3e96d9092486a6f683234e15e9434ff8036497c67: read blob <data/4aaa51c6> from 4a6e87af: wrong data returned, hash is 756235576898f59c0852114d007dbe9dce2625bd1b0f2a018d78eabb7105d2f2] [2:41:33] 100.00% 54703 / 54703 packs
The repository is damaged and must be repaired. Please follow the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html .
Fatal: repository contains errors 22:49:06user@backup:~$ su - Password: 23:01:28root@backup:~# restic check --read-data -r /path/to/restic_backup/ using temporary cache in /tmp/restic-check-cache-722510635 create exclusive lock for repository enter password for repository: repository 4336eb3a opened (version 2, compression level auto) created new cache in /tmp/restic-check-cache-722510635 load indexes [0:01] 100.00% 31 / 31 index files loaded check all packs check snapshots, trees and blobs [0:02] 100.00% 16 / 16 snapshots read all data [2:39:48] 100.00% 54703 / 54703 packs no errors were found
19:27:41root@backup:~# restic check --read-data -r /path/to/restic_backup/ using temporary cache in /tmp/restic-check-cache-629501794 create exclusive lock for repository enter password for repository: repository 4336eb3a opened (version 2, compression level auto) created new cache in /tmp/restic-check-cache-629501794 load indexes [0:01] 100.00% 36 / 36 index files loaded check all packs check snapshots, trees and blobs [0:03] 100.00% 24 / 24 snapshots read all data check successful on second attempt, original error pack f5f16acb02e8fe233fa3fc47171ff0a4db7fbb60241d886bfec3f844c0823a02 contains 1 errors: [blob 50b167aea09499d969b3e86076fff32688acd663393d217c2665c423fdaee8e2: read blob <data/50b167ae> from f5f16acb: wrong data returned, hash is 759553a8995c07ae9603c1b91cbe12478da246285c0cd88d3fec7ef6d4e8a390] [2:42:26] 100.00% 54716 / 54716 packs
The repository is damaged and must be repaired. Please follow the troubleshooting guide at https://restic.readthedocs.io/en/stable/077_troubleshooting.html .
Fatal: repository contains errors
The behavior in the logs is the very definition of bitflips. I unfortunately don't have a good idea how to properly diagnose this any further.