Recurrent cache reading errors when pruning

[Chostakovitch]

Hi there,

Launching the prune command randomly produces this kind of warnings:

rustic -P <profile> prune
[...]
[WARN] Error in cache backend reading Pack,f3dcf853: Error: Failed to read at offset `3387397` from file at `/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f3dcf853319d722beea443c9d766f1c4b708dfdfe9c1dd41e5b119aafd3a493f` (kind: related to input/output operations): caused by: failed to fill whole buffer

Launching the prune command again does not.

Is this a "normal" thing to happen? Can I do something to mitigate the issue?

Please tell me if you need more debug information.

Thanks in advance!

[aawsome]

Hi @Chostakovitch! It seem that the file in your cache is too short (for whatever reason). Can you run rustic check to see if this is the only problem?

If yes, you can safely remove the file /root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f3dcf853319d722beea443c9d766f1c4b708dfdfe9c1dd41e5b119aafd3a493f (or even the whole cache) and this error should be gone.

[aawsome]

Maybe we should add a rustic repair cache command which is able to do this automatically... But IMO this is not the highest prio. Cache files are really exactly this - they improve performance but can be safely removed at any time.

[Chostakovitch]

Thanks for answering! I didn't realized that even if I got dozens of warnings each night, it always points the same file.

However, rustic check reports nothing wrong and the file does not exists.

I'll post another message if it stills happen. Could it be some kind of dangling reference ?

[Chostakovitch]

@aawsome I didn't managed to solve the problem yet.

Is it possible to reset the cache ?

[aawsome]

Is it possible to reset the cache ?

Simply remove the cache dir, it will then be recreated the next time you run rustic.

[Chostakovitch]

@aawsome even after removing the cache, I got dozens of those errors during the next backup operations.

Do you think those can be ignored safely (and this issue closed) or could it mean that something is wrong with my data/repository ?

[aawsome]

@Chostakovitch You can try to run a rustic check --no-cache to see if there are problems in the repository while not using any cache. If there are no warnings without using a cache but still the warnings with the cache (rustic check without --no-cache), it seems that there might be problems with the place you are storing your cache. Missing permissions? Problems with the storage? Can you please describe where your cache is located and try to read and open some files in the cache dir?

[Chostakovitch]

@aawsome thanks.

Checks

check gives no error, with or without cache:

rustic -P global check --no-cache --log-level=debug
[INFO] using config /etc/rustic/global.toml
[INFO] repository local:/data/backup/repository: password is correct.
[INFO] using no cache
[00:00:00] getting snapshots...           ████████████████████████████████████████        487/487                                                                                                                    
[00:00:00] reading index...               ████████████████████████████████████████         24/24                                                                                                                     
[00:00:00] listing packs...                                                                                                                                                                                          
[00:00:06] checking trees...              ████████████████████████████████████████        487/487

---

rustic -P global check --log-level=debug
[INFO] using config /etc/rustic/global.toml
[INFO] repository local:/data/backup/repository: password is correct.
[INFO] using cache at /root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679
[00:00:00] getting snapshots...           ████████████████████████████████████████        487/487                                                                                                                    
[00:00:00] checking Snapshot in cache...  ████████████████████████████████████████ 190.35 KiB/190.35 KiB 44.23 MiB/s  (ETA -)                                                                                        
[00:00:00] checking Index in cache...     ████████████████████████████████████████  54.25 MiB/54.25 MiB  3.50 GiB/s   (ETA -)                                                                                        
[00:00:00] reading index...               ████████████████████████████████████████         24/24                                                                                                                     
[00:00:00] listing packs...                                                                                                                                                                                          [00:00:00] cleaning up packs from cache...                                                                                                                                                                           [00:00:00] checking packs in cache...     ████████████████████████████████████████ 270.81 MiB/270.81 MiB 3.20 GiB/s   (ETA -)                                                                                        
[00:00:05] checking trees...              ████████████████████████████████████████        487/487

What's strange is that I cannot reproduce the cache issue reliably, i.e. running the whole backup/prune sequence sometimes produces an error, sometimes not.

Configuration

Backups are triggered at night with a systemd timer, and the following service configuration:

[Service]
Type=oneshot
# backup locally
ExecStart=rustic -P %i backup
# forget old snapshots and mark unused data for deletion
ExecStartPost=rustic -P %i forget --prune
# check snapshots health
ExecStartPost=rustic -P %i check
# save on remote server
ExecStartPost=rustic -P %i copy --init

Note: ExecStartPost commands are executed sequentially.

Cache errors always happen at the forget --prune stage.

systemd units are parametrized with a profile name. The scenari profile, for example, looks like (minus hooks):

[global]
use-profiles = ["global"]

[snapshot-filter]
# so that commands are automatically narrowed, inc. copy
filter-tags = ["scenari"]

[backup]
tags = ["scenari", "timer"]
globs = ["!/**/.git/"]

[[backup.snapshots]]
sources = [ 
    "/var/lib/docker/volumes/scenari",
]

Note: this means the backup sequence can be triggered manually with systemctl start [email protected].

The global profile, included in every profile, looks like (minus opentelemetry):

[repository]
repository = "/data/backup/repository"
password = "<REDACTED>"

[copy]
targets = ["rest_server"]

[forget]
# keep less snapshots locally, only the most recent ones
# see rest_server.toml
keep-hourly = 1
keep-daily = 24
keep-weekly = 7

Note: as you can see there is an external rest server which, imo, is out of scope.

Additional information

1. All commands are run as root, so I don't think there is any permission issue.
2. Similar errors happen on every machine running backups. They run different services (so different data) but with the same configuration skeleton.
3. Cache is located (on the main machine) at /root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679.

I can browse the cache (excerpt below):

ls -R /root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679
.:
data  index  snapshots

./data:
00  03	07  0d	14  19	1c  21	27  2d	33  38	3e  43	46  4b	50  53	59  5f	64  6a	71  77	7f  85	8b  94	9b  9e	a3  ad	b7  be	c2  c7	cc  d7	de  e5	eb  f0	f8
01  04	09  11	16  1a	1d  23	28  31	36  3a	40  44	48  4c	51  56	5a  60	65  6e	74  78	81  87	8f  95	9c  a1	aa  b0	b8  bf	c4  c8	d4  da	df  e9	ec  f3	f9
02  06	0c  12	18  1b	20  24	2c  32	37  3c	41  45	4a  4d	52  57	5d  63	67  6f	76  7d	83  89	90  96	9d  a2	ab  b5	ba  c1	c5  ca	d6  dc	e2  ea	ee  f7	fd

./data/00:

./data/01:

./data/02:
02bf1d760494a1481520ab9d0a3d5f92887defe1f26cb261f5101807562c4857

./data/03:
03018fd0ea5f5ab941a722c64cce00f948150fdfcb71ed477018a9c3fbd9af76

./data/04:
0472758c0d922fcd6748a64964376ac1ef0e9aa2fa37e2a2b7138c4d8e265050

./data/06:
067bc6a0cbc263e523c8782857d02bf23fcfddf4bc9751b8200332cc93a2256e  06fa6934cef8ed93b802dda7b87f85c7836308dfdda7d6a890d94cb53e13d849

[...]

./index:
02  09	0d  15	1e  23	27  2b	2e  36	3a  3e	42  4b	51  59	5e  67	6e  76	7f  86	91  9a	a0  a4	a9  ad	b3  b9	be  c3	c6  c9	cd  d3	d8  dd	e4  ea	ef  f7	fd
04  0a	0e  19	20  24	28  2c	2f  37	3c  3f	47  4d	53  5b	60  6b	70  77	80  87	92  9d	a2  a5	ab  af	b5  bc	c0  c4	c7  cb	ce  d4	da  e1	e7  ed	f0  f8	fe
06  0b	12  1d	22  26	29  2d	31  39	3d  40	4a  4e	55  5d	64  6d	74  7b	81  88	94  9e	a3  a8	ac  b0	b8  bd	c2  c5	c8  cc	cf  d6	db  e2	e8  ee	f3  f9	ff

./index/02:

./index/04:
04c48d8e2ff6b52457061258564577ceee27313a350dec5be58f5ae3c74cc837

./index/06:
06a06e36aa20c482a0baf916dc112d490a355ebe900ceda62661c2727ae1e712

./index/09:

./index/0a:

[...]

./snapshots:
00  05	0a  10	15  1c	24  2a	31  3a	40  45	4a  52	57  5c	61  66	6c  72	77  7c	83  88	8d  96	9b  a0	a6  ab	b0  b5	bb  c1	c6  cc	d2  d8	e1  e6	eb  f0	f5  fb
01  06	0b  11	17  1d	26  2c	33  3b	41  46	4b  53	58  5d	62  67	6d  73	78  7d	84  89	8f  97	9c  a1	a7  ac	b1  b6	bc  c2	c7  cd	d3  da	e2  e7	ec  f1	f7  fc
02  07	0c  12	18  1e	27  2d	34  3c	42  47	4c  54	59  5e	63  68	6e  74	79  7f	85  8a	90  98	9d  a2	a8  ad	b2  b8	bd  c3	c8  cf	d4  dc	e3  e8	ed  f2	f8  fd
03  08	0d  13	1a  1f	28  2e	35  3e	43  48	50  55	5a  5f	64  6a	70  75	7a  80	86  8b	91  99	9e  a4	a9  ae	b3  b9	be  c4	c9  d0	d5  dd	e4  e9	ee  f3	f9  fe
04  09	0e  14	1b  22	29  30	39  3f	44  49	51  56	5b  60	65  6b	71  76	7b  81	87  8c	95  9a	9f  a5	aa  af	b4  ba	bf  c5	cb  d1	d7  de	e5  ea	ef  f4	fa  ff

./snapshots/00:
003d0d072e0d89c196ff4c743b870040a4d4e688acfa2e0801a0b17e25bb5769  0042ac5a83fe2a7793648546dbbbdbe579bcd1c8f6fb67faa8bd908662362a72  008c1ab7602648a015845c59098397026422437355c9d0299d654676419f0d22

./snapshots/01:
01133f4ab0b43f970367b9f5c2bb7448bf0393b0bd7f6e4c321411cb9f6b32c8

./snapshots/02:
02084e9fa38070d26fd63ade5eb024426a5874d12a5ad711da73248069b3b620  0214018bb5de3d634f2e714ca3754fc3d874a900bcb01b051d0d10d19e4ebf22  028d672e4b3010fab00764a650df87546adcbcae0bd4ab28538c47f9fa545e6c

./snapshots/03:
0392a10892e85344ae627a0c29bfaff6fdcb5c5ff814b052c1cb5fc8a6c8dd1c

./snapshots/04:
04724ed1789a97994d3c8671f75b162f154c46bd543a3806ababa795b4df5c4a  04c7ad749a62b10e8eab93978b7ca6ab8fdd4a6de16b140be96d4aaa8a7da0cc  04eac834a8e8580de23bee75df69b229cfa8b4dd1bcd18b0d6a74e172f0025f5

./snapshots/05:
0516f43c10ba697f48f3a946475d3b855ecae09da6b482f5ed56c9da82790a61  054523f12ca32b9a25d9356c8f73786f1130f9c089c2398826ecacbb5c868c34  05d357b5d4bb653abb3f5aa0ffaeb446915cca829702003a3e81b6fd4f06b8b1

./snapshots/06:
06153fa4f2ca5707005fde1f201073d9f55502a13106281f2607223e0746034d  06cec7fa0649a008597453989516a9e0dbe73b9e95d3d6ebcc92cf0207ac3fd2

./snapshots/07:
0728c0a2faaeb15d6a36bbd377753699824fad9703fb7c00f49cc694bf73013c  07b465eb1c16bb2850c15039b890ad67dd8fcc48d90a5cb42c7b5e67644942d9
0766ce436276f91dcf0a9a8d6dd5c31430719c93c7d10e43dca2f0c3706d95af  07ceeeb3455a3508dc6b1164b855e942d5b0f7ee9444b5a17f1adb7e3137edbd

./snapshots/08:
085a248d012cd947bd8fa3fea256b4565447e17c9cc9de4a5a3f2099e2ec7130

./snapshots/09:
09b08d7f183d7fc2335abf7938c6c5f7e264ca94054551c25f1383dc9166b510

./snapshots/0a:
0aaa7be90b6e4c22a2b8ac36b5d8fe38f2e7ccd6bef815261326ef927b6bda00

./snapshots/0b:
0b6e30b5d8557a9378b4b5f9a74b95169643194c85ef17494dd39cbdcc522d4a  0bb3fdda982aaec32d9212f2e1705a95a188893da5b5f9615988058781cc3946

[...]

Every file seem to be readable; most of them have a size of 4096 bytes, some are bigger. Excerpt below (hidden lines are all 4096 bytes):

find . -type f -exec du -sh {} \;
2.5M	./index/06/06a06e36aa20c482a0baf916dc112d490a355ebe900ceda62661c2727ae1e712
2.5M	./index/d4/d48d76c3c12498739086ad193e3cb874e25e6745742f1ea4fdb7c7014bfb62e1
2.8M	./index/be/be827b6c0bf06d00f7216d98216522de236f1be18f150676dcd77786ce8fd793
2.5M	./index/ea/eac58f48e59d9bd3e2c526a4c6c7a2a274b3bf827aec305de58241f1ba23c4f4
4.0K	./index/5d/5df99933691a61ff4680a60370f0a89597742d51952cd6acb541e3d40bc22c9e
2.0M	./index/d3/d3b769fecdc01ef7e88eb18996cc88ab7f846171540352d2e4ab81eef7fcf2f0
2.5M	./index/36/36690d6d4046c33c6b92fbb5871b95cd94506223ca891503e108aa33f307d60e
2.4M	./index/6d/6dab8aa0c2ca5bc6c1c6653ba8cf32ecab4d52504614b6491359503612e4c70c
2.5M	./index/5e/5e0872ebde501147fe07ca18562731f362b1d3c5df6b815115797cb8bae03885
2.5M	./index/53/53348683970656764ed48e69fbf5a97656335325d76ed16e3ff38cac56571f35
1.0M	./index/f3/f3159afbad237d0dc63fea40bbffe13c5373422d85f32ec3cc6bb3549aae2979
2.5M	./index/b5/b510808029a0a186f38ff3bf38099542a534236a7c47ae1e7e9d8a90906b866f
2.5M	./index/94/945f6eb7c48ad4e5d6e926dcb3abc2a12b35f37bc8f60bb79c9c107ac5dca8b4
2.7M	./index/f0/f0be380b2afae041bf1258fa6a7b67c1aa0cebba2d8d5398c9c64a3de7009864
2.5M	./index/3d/3db9a10fdf144eadc30ada8c91f8806865c418576c35a8cf41da1701217cc3c4
2.4M	./index/c2/c2b83e4d129460e6e1d0bd16c43b73edb9410d79114c0397e743e92dd9954765
2.4M	./index/24/246b2e4f779d357d3bb4262629d770ad154a86bb847493b6fa1be90abb994d21
2.5M	./index/12/12574457bedc1da581352ddf5c2a03cdcd906725397294bfb3f467e016053ff9
2.5M	./index/a3/a35ceb6212f2822573fe63f87718c7312c1ffcdf1f9a03d26d356050988630f4
2.4M	./index/a3/a39ae29d3596334679639ae00f879618e57aa01b163e80fff228e5af4dd92b1c
2.5M	./index/6e/6e0236e441b0bb44b4cb6115cbb9e7fad84464522fae93d9ac9b1dd21ba21a6c
2.5M	./index/04/04c48d8e2ff6b52457061258564577ceee27313a350dec5be58f5ae3c74cc837
2.4M	./index/ed/ed4ff6ce90ba86b4e2d4da05c041ac15459c35f6e2c2a845216d68edb92a9459
2.5M	./index/cd/cda871ac9667dba54fdb07e72585ff6e2ba457892758867d274b58fa86efaed0
4.6M	./data/83/83ba9c87a603be3a66fdac7719ae7b5e6f8416c0b4c01763b2a4f5cbe4610873
1.6M	./data/52/52239ae40a46de4cd0a18836fc5b885dd5f554bfba8b753dfc5c8c5e6682315d
1.6M	./data/06/06fa6934cef8ed93b802dda7b87f85c7836308dfdda7d6a890d94cb53e13d849
1.6M	./data/06/067bc6a0cbc263e523c8782857d02bf23fcfddf4bc9751b8200332cc93a2256e
4.9M	./data/d4/d4a0a0a2f6bd9ee90f9214a75e17deeb6b41881a0b8d31a7cd337b222fa5d350
18M	    ./data/71/71a267f324c6a331d4d60ac3ba6fb4e6e51e483541e2282b0311eebfa87a43fc
2.0M	./data/71/71ba2e1a897ea4a1cb81057bec2c2a3650a2cc2096539a18a6d1f3499590697e
4.7M	./data/f9/f955e45ddbb1877f40b0441c52ee8a80678fd21efb9f8a3742e46894f8227772
2.2M	./data/85/85c97836b4a08832ca77dffdb65f9a4a78676ff4670efd33ac4b7361b9840ea4
4.9M	./data/ba/ba5fde6afb53d0bdf612329cf28b2c01e376429dcb823a49d0fcca526a990a5c
4.6M	./data/ea/eae1c9e76b805fc03a1ae6b8d618585ad7b58cc381c5b026da8763bd23b8b7c0
2.0M	./data/2c/2c5d5c69d573752762516d549277823f30bc967cfe40e1e4fafd1fb8abe85730
4.7M	./data/5d/5d93b2f1c05ee9a60e299fba7e1aced154ca0241c27b94fd2e8cc501f7a7756d
17M	    ./data/59/59a14e03aeeb1cb31bb8d2862986f68fcffa79ce33e7ad6eec742a2ce7f8a41c
1.5M	./data/87/8703e3e328cc52ade1546237f79b1f97e90c40a154e101943224ab8acc796f59
17M	    ./data/1a/1a1e8918e0b4257a8436e4c0ceebd75808fc472908c170e6c6629b428c1ed255
1.9M	./data/36/365566001cb912deb9bec7b30f64d4a7672b943d27d75f9800f5ff960dfa9d20
2.8M	./data/36/36b989ca0c8157904f573bf48a719d07ef9aebc6e815f35fa09d6250e698e134
6.6M	./data/09/09c5bf9675ea103d1b160b45982acda41ab1a29bc70131f914bfecfafae4a677
5.1M	./data/3c/3cef6ed7fcac9f6cbd182bc838b7f5c6a0abbe8868e459643c1236e263fc3f49
15M	    ./data/28/2811b7cb0d79a96094e6e8aa7ca1e20982e29065982f22164c7dbb9635bdd9d3
1.6M	./data/76/76463b7d100e9cca0f576d5783c9df07b99ba3ffe9dc47042e04335c51f495ef
4.8M	./data/a1/a1861f2eab1fe62e6f04b89b96aa3f2d99b02cb1c41af1f7c28ac50323fb29ec
2.1M	./data/03/03018fd0ea5f5ab941a722c64cce00f948150fdfcb71ed477018a9c3fbd9af76
1.6M	./data/9c/9c44b06dc07d66cf339d2c6d7966a960498bddc9a6b698692afae82699455615
1.6M	./data/6a/6ac17f7bca70214d71f2067f4e673224a487c91d6cf42aa42ea18c79d62265c5
1.5M	./data/14/1469d870ff57ef08be9c5c1226dad6fd4c198673f9dc1c00e6895310a435a96d
1.5M	./data/14/14f02140fa318902a1900be74ccccc5947841d70fc47cdbca96132706503c518
1.5M	./data/4d/4dbce49bc615eef9f29e83e2feee40945d58f5f3ba861160ec72bd80bcbe4ba2
4.7M	./data/78/782e14000fdb164dc52706414d5ad9353c150a372c15f9b85c05e325fa3ac15d
56K	    ./data/b7/b7b2e597ac963825e7b3724061f3cea2ca5f04deefffa9a368eaa715dc15696f
1.6M	./data/7d/7d7b3cbd5f983c8eacb818f0a3412cf0d92fe64cd13f82c3f5c01d86faeba99b
2.1M	./data/f3/f33d0be15b280e58cedadaefdcb4fcfc3afbb5edf6c85e10cf0f6a69b89032ae
17M	    ./data/f3/f363e33eabcf6db7fd948cf7d49eace415b8d979314ec65f8b6ba6299fdfedb1
4.4M	./data/f3/f3bbb2462ec628c1b6621a0bdf4947fd06c85127c7f886a85f699e898a0720c9
1.5M	./data/c7/c7fc1cb27faaf9f9851e79ba4da0e5b0c2eff3da6c0451950c71216d9716cbff
4.7M	./data/ad/ad128969383d90090f368e93507a453d809083de0d383239a1e28f91e9d8e26f
2.2M	./data/ad/ad0f609b92a0f4750151260c19fe3d670de478d64951fcc74846098d6c3f402e
1.5M	./data/94/94ef38b9b9a6ba9b03785cb50dcf73160c718769657d149b0feac2d58e0101ac
17M	    ./data/8b/8b5b0176a638feaded1bc12e3a86ec324bdc0cf1be213386a7eb79c2e13a1092
16K	    ./data/df/df2d382e6acdffc6f1b738e1139e1e2621423ccafcbbe0b0d5ad6d6319f5e5be
4.8M	./data/31/3163c1fd4914d6b0125362599c131007151af98cbd220338efd15c6b38205ce1
1.8M	./data/02/02bf1d760494a1481520ab9d0a3d5f92887defe1f26cb261f5101807562c4857
1.5M	./data/4a/4ada2c72be8f526e8389586f5ebc129a6b870df14296bcae032b7bdbee1e6e66
5.2M	./data/48/48783213deb11061ad39f3f6627e3ac986c4938c07600e5f3502b044bc53db6d
2.1M	./data/12/12d94ac8de0005c26e7b08d6a2fa89a7982dba9af6fe678a868a3435ca54d4a5
2.1M	./data/5f/5ff48249bcfb283f33b1ed5ccc6bbfb0f213c77dc314061d1cb034615e5050c8
4.8M	./data/5f/5f69d031876e78430b6d964699a7a709661d0c20ad4569a3b25ec1cfcfa1fe7a
4.0K	./data/43/43e1c7119a3644739ca85f9ba4a8c96e1c264666d42ed6063f945353ac127e59
292K	./data/60/6005ff6311322c91f187f7024e53b939198c6f6e29225673c43ff2aa590c5fc4
372K	./data/a3/a3e126cc54d604372572e7c1d49e6a4c5b375ed02c967ea19ed554cf560c2680
4.6M	./data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
2.0M	./data/04/0472758c0d922fcd6748a64964376ac1ef0e9aa2fa37e2a2b7138c4d8e265050
1.6M	./data/aa/aa60468115179457c8f9a0689b30c50e440986bb44bc8aa911195f02aaa8d326
4.7M	./data/65/65c0349f5320f9bea0148acf070e15a61d526a268616bb964a629f4650c84510
12K	    ./data/46/46478bbe1828b60d62c74d26e76a4a5b6293fc23f6c47424818d2a00afa3f01d
17M	    ./data/6f/6fe5cae5a26d509b28b30fdfd79a0dd912e7774c2a413d68e99b727023cc31de
4.7M	./data/9e/9e48b4f3fdcc01d98e948195e3ed49509bce9f7e095a7f6fccb520d1fcb1b598
4.8M	./data/9e/9e1e00ed663ba9749fe24d8f6fd236498b0f327c560392bba8738a4671c14d79
17M	    ./data/63/638cc44860bf4bac3ed99bcb8e22b5e97e3bb12d7d0e3fbed59f3debbe3b5791
4.0K	./snapshots/83/83b7baa6ed81a9376b5d847bbc6d084971887a10ffd426b220fea27bbbd39318
4.0K	./snapshots/83/83611924aec76648dc415012cf372a1da10e42b8bdf14f11e520b604bfebd99d
4.0K	./snapshots/83/8353f7476cf0184622e847c60cc9d09f1e48be66d5ac095af2424804789fe43e
4.0K	./snapshots/a8/a84b4f762911eec2bca380e3050edcc38e12f00a9b3268a7c96362126e3feec8
4.0K	./snapshots/a8/a8addb4088ebfeae61f883df4f4f437008543ee07e3b7afa64d57f0889e66a94
4.0K	./snapshots/b4/b4b1fe8c379e6f30798df0c48a185dd1cfc7618348b6ca5007e3b3fa09c8b896
4.0K	./snapshots/b4/b4c5e6d32abfbe6035d790b7dedb01db8e9ff84831ac8d5ceb94da6c3e10e5a7
4.0K	./snapshots/52/524fb99bd48c3f986f2b363cd7a740209a71fe83cdbe6479b02e8be63d5e1a9b
4.0K	./snapshots/52/52b04996c961f6d425b465ba0c64c0f05093e0855b491438ab9f6fbe002d3d09
4.0K	./snapshots/52/52f298d30d3453f09944e3d29c9eb2e3bdf6f3d5aff3db66c9e56c4edfb39769
4.0K	./snapshots/52/52314ca057721b879b310cd5fa873aa0c8e51c933988862c8164a794180c353e
4.0K	./snapshots/97/9726cc5940a25d71c09124a59a2ee5dd864bf36129f1f47e617dc19670b159e5
4.0K	./snapshots/97/97799fc5879f4373862cc6bf64cd15a948e7a013f727f330df96ccc73c6642dd
4.0K	./snapshots/97/977b49e962e48492dc6876e4a42c5b9a0ea6a65a6684a5c3a9728543104490e7
4.0K	./snapshots/84/8402ed189c3edba662bddc731a7c2dffb192ff0bf08244ddb6947ff0eb2e3e48
[...]

A closer look to the files pointed by errors

Here are the files pointed by errors on the main machine. Errors are since I deleted the whole cache 2 days ago. Some of them exist, some of them do not (is this normal?):

sudo journalctl --since "2025-09-29 00:00:00" --namespace=rustic --no-pager | grep --color -i error | grep -Po '(?<=`)/root/.*(?=`)' | sudo xargs ls 
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/44/4441e6e750b9e0d8915f3cbd9471046a81342fd17983ac96afa7d4a7ec993df4': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/21/21dc497f949675a36dac0ae46cb438abc537718ff2263b3a2e5055833daa1026': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/21/21dc497f949675a36dac0ae46cb438abc537718ff2263b3a2e5055833daa1026': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/21/21dc497f949675a36dac0ae46cb438abc537718ff2263b3a2e5055833daa1026': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/07/0743b9b559e7b19f4dddca2e00c6865ec795c15b77c76554fdce417be82a0770': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/2d/2da7f0515b91983d50fc3bc5eb8641878420cab314292adfb33eb5cb97b28d14': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/de/de498600b54aeb68421c6e0482fe7f747f8089020eb94e654e062e562243a422': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/de/de498600b54aeb68421c6e0482fe7f747f8089020eb94e654e062e562243a422': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/de/de498600b54aeb68421c6e0482fe7f747f8089020eb94e654e062e562243a422': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/de/de498600b54aeb68421c6e0482fe7f747f8089020eb94e654e062e562243a422': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/de/de498600b54aeb68421c6e0482fe7f747f8089020eb94e654e062e562243a422': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/de/de498600b54aeb68421c6e0482fe7f747f8089020eb94e654e062e562243a422': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/2d/2d4c89061b67978b152f36570dbff6592261eeeecb3bd71a4a2682d6d82351ab': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/1c/1cc54101b641e093474aadfacc2e8c60ef34291da3770fdaee5b099efbd8ad7a': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/c7/c72fa0fda5cd6a766a4c479157b9f5171c15a9d2d92f20c990bf4d5e1109f231': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f34b28c908249b6385a0681523de0ca8ddd79d509f3fd3454cdb1ff85938ff2b': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f34b28c908249b6385a0681523de0ca8ddd79d509f3fd3454cdb1ff85938ff2b': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/e9/e994f036bded81f266d608bd9e37e4406606da48e173cd8d150c60b7ef94ba7f': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/32/32c6a331cc27f3c0c33e8c058f95ab67574fd92b635629dff6a432e2fdf236ed': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/c2/c2122940d15eabb934db7b095976af8655ab3457e8e54e37287b6ffc83466a33': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/38/386508fc7051f95a974fa4118ce2d47597fb4ba404116cab2134ad30862b79bc': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/01/01c74cd6153ef1aaa74cd1c4f7588b20a8a514b200736248e91d6678e4c5ea7d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5d/5d6e78c8d9a4b981d150f23b3d9710c30157a4d5241deec208610926d0cd0242': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/01/01c74cd6153ef1aaa74cd1c4f7588b20a8a514b200736248e91d6678e4c5ea7d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/e2/e2a49fee239bf5b3a2449c87555028126447e315bca7a99a21166c6a0d80345d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/5a/5ac33119e2fd496d53a1126bb6cbd044262a05c382ba457a7932e3121e6ee30b': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/77/777eb335934395603578a4727393c6d68516221391548c8444413218ab34af42': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/dc/dca880f056c800e2f1476ad87f7f04d238b3b3912e62ed34600c428d8cd51b6e': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/dc/dca880f056c800e2f1476ad87f7f04d238b3b3912e62ed34600c428d8cd51b6e': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/c1/c16d41530d61838632b64f7e751bf5f6f19e51e236f22cc95b55d7399bb27c4f': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f30b381516ad3fc6ca68221690f15409da1cb0b7b14496b62dfd4fefa075324d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f30b381516ad3fc6ca68221690f15409da1cb0b7b14496b62dfd4fefa075324d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f30b381516ad3fc6ca68221690f15409da1cb0b7b14496b62dfd4fefa075324d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/53/5387f3f788ac245e05434b67779de945a156fe67a67a066faa46e958a407211f': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f30b381516ad3fc6ca68221690f15409da1cb0b7b14496b62dfd4fefa075324d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f30b381516ad3fc6ca68221690f15409da1cb0b7b14496b62dfd4fefa075324d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f30b381516ad3fc6ca68221690f15409da1cb0b7b14496b62dfd4fefa075324d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f30b381516ad3fc6ca68221690f15409da1cb0b7b14496b62dfd4fefa075324d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/f3/f30b381516ad3fc6ca68221690f15409da1cb0b7b14496b62dfd4fefa075324d': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/4a/4a2bd33a55c30ec953dbe4f216ddba44e59aa50187ac02fe5c12b21fbb4d848c': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/c8/c891058e0ca6d74a0e04616e034d363e220ca624126e0b39ee8382bdc7d6a7b7': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/9d/9d097d2d6b3ed15713b3493637a6aef42904539528772abee9aabb430d0a1eca': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/2d/2ddd01cd497456ab4c1b8ef99fd561b633ed40828fb7a27ffd8c37a763f3a54f': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/18/180d57a6d1fa5f3dcd5c814df70c6d9c0fbe7824b003996b4d4dce2fd8d60719': No such file or directory
ls: cannot access '/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/9b/9b56c93a138a1a748b85656eb75225d839fa6d37cf65e386028330bc710767fe': No such file or directory
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/59/59a14e03aeeb1cb31bb8d2862986f68fcffa79ce33e7ad6eec742a2ce7f8a41c
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/59/59a14e03aeeb1cb31bb8d2862986f68fcffa79ce33e7ad6eec742a2ce7f8a41c
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/59/59a14e03aeeb1cb31bb8d2862986f68fcffa79ce33e7ad6eec742a2ce7f8a41c
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75

The last file, which still exists, is involved in multiple errors, e.g.:

Sep 29 00:57:50 scenari rustic[1255156]: [WARN] Error in cache backend reading Pack,6ec3feae: Error: Failed to read at offset `3543824` from file at `/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75` (kind: related to input/output operations): caused by: failed to fill whole buffer

Here are some information about it:

sudo du -sh /root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
4.6M	/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
sudo file /root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75: data

The file seems to be readable at whole (output truncated):

sudo hexdump -C /root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/6e/6ec3feaed6e96dc1f3cc4a222b640b11dcbbda750a32c9075c2bab490e2f2a75
00000000  ca 30 1f 84 01 87 31 e7  c0 c2 7a ea 81 40 fe 5a  |[email protected]|
00000010  a6 d8 ac 49 12 da 7d a1  4a 23 78 1f a1 44 ee 28  |...I..}.J#x..D.(|
00000020  b1 03 ce dc 4f 83 0e de  39 ba b4 5b 35 ba 17 d2  |....O...9..[5...|
00000030  84 b8 9f 83 f0 4a 81 ae  f3 b6 8c 23 87 34 51 dc  |.....J.....#.4Q.|
00000040  da b7 d5 5b 64 c1 98 ef  db ca 50 60 98 2b 07 18  |...[d.....P`.+..|
00000050  58 ff fb 0d e3 87 40 a2  15 56 54 65 d9 e1 c8 1d  |[email protected]....|
00000060  b4 fb 96 e3 c5 9b 23 ab  57 66 68 ea dd c8 b8 7d  |......#.Wfh....}|
00000070  b9 ec 4b 70 7b 43 92 9e  ca 9b 24 63 a0 cf b6 7a  |..Kp{C....$c...z|
00000080  18 3c 07 e7 c3 fc 3e aa  05 28 0e bb 20 9f aa bf  |.<....>..(.. ...|
00000090  70 76 fc 05 a6 b3 ff 20  ac c3 71 4e 05 87 28 e0  |pv..... ..qN..(.|
000000a0  00 3d 0f 14 8a 88 fd e1  15 56 11 34 bf 70 c1 1c  |.=.......V.4.p..|
000000b0  70 3c cd 41 46 00 fe c5  8b fe 82 02 17 d1 83 cd  |p<.AF...........|
000000c0  a8 86 eb c3 f8 9e a0 e1  a7 73 e3 45 2c b9 3b 9e  |.........s.E,.;.|
000000d0  ad f5 53 0e 9c 53 7f a9  fc 95 f2 dd 1f 31 4a 10  |..S..S.......1J.|
000000e0  b7 3c 3f 9d 00 05 6d 34  db f7 70 6a 2e 4d 20 7e  |.<?...m4..pj.M ~|
[...]

Finally, there is a lot of free storage space:

sudo df -h
Filesystem      Size  Used Avail Use% Mounted on
udev             16G     0   16G   0% /dev
tmpfs           3.2G  3.6M  3.2G   1% /run
/dev/md3        1.8T  539G  1.2T  31% /
tmpfs            16G     0   16G   0% /dev/shm
efivarfs        496K   20K  472K   4% /sys/firmware/efi/efivars
tmpfs           5.0M     0  5.0M   0% /run/lock
/dev/md2        988M  141M  780M  16% /boot
/dev/sdb1       511M  4.5M  507M   1% /boot/efi
tmpfs            16G  1.6G   14G  11% /tmp
[...]
# Here are docker overlay2 mounts

A dead-end guess

Reading this StackOverflow thread, I wondered if those errors could happen because some forget --prune commands are run in parallel, and that somehow the same blob is written and read at the same time?

Backup time are randomized on a two hour window, but I can see that some of them have run in parallel last night (scenari and mail):

systemctl list-timers --all
NEXT                                  LEFT LAST                              PASSED UNIT                                             ACTIVATES                                         
Wed 2025-10-01 12:00:00 UTC       2h 18min Tue 2025-09-30 12:00:00 UTC      21h ago rustic_prune.timer                               rustic_prune.service
Thu 2025-10-02 00:56:45 UTC            15h Wed 2025-10-01 00:56:45 UTC       8h ago [email protected]                           [email protected]
Thu 2025-10-02 00:56:55 UTC            15h Wed 2025-10-01 00:56:55 UTC       8h ago [email protected]                           [email protected]
Thu 2025-10-02 01:18:21 UTC            15h Wed 2025-10-01 01:18:21 UTC       8h ago [email protected]                            [email protected]
Thu 2025-10-02 02:09:23 UTC            16h Wed 2025-10-01 02:09:23 UTC       7h ago [email protected]                             [email protected]
Thu 2025-10-02 02:35:27 UTC            16h Wed 2025-10-01 02:35:27 UTC       7h ago [email protected]                                [email protected]
Thu 2025-10-02 02:35:50 UTC            16h Wed 2025-10-01 02:35:50 UTC       7h ago [email protected]                             [email protected]
Thu 2025-10-02 02:50:47 UTC            17h Wed 2025-10-01 02:50:47 UTC       6h ago [email protected]                            [email protected]

However, a cache error occurred last night for platine, which ran before the "parallel backups":

sudo journalctl --since "2025-10-01 00:00:00" --namespace rustic -u [email protected] --no-pager | grep -i error
Oct 01 02:09:33 scenari rustic[3206462]: [WARN] Error in cache backend reading Pack,9b56c93a: Error: Failed to read at offset `2148478` from file at `/root/.cache/rustic/822c46af18f258ff765facec47fd66022f13e7f573e9c5400ea321955aba2679/data/9b/9b56c93a138a1a748b85656eb75225d839fa6d37cf65e386028330bc710767fe` (kind: related to input/output operations): caused by: failed to fill whole buffer

Last words

Please tell me if you need more information, and thanks again for your time.

Maybe at the end this issue is harmless, as I can restore and/or browse snapshots. But if you think it is abnormal and/or mask another problem in the code, I am willing to help narrowing the cause.

Have a nice day!