rustic
rustic copied to clipboard
rustic-rs
rustic `check` needs a lot system cpu
rustic check run on a 64TB repo gets killed with OOM.
- checking trees...
[00:29:17] █████████████████████████████████████░░░ 4479/4751
Killed
@borkd Thanks for reporting!
Now, this is an issue I'm very much interested in fixing as one of rustic's promise is to use less memory than restic. However, I have never tried it on such an large repository.
There might be some non-ideal algorithms (most likely the one reading all trees) which use too much memory in your situation. I'll have to take a look and think about it.
Can you determine how much memory rustic uses here? And how much restic for comparison?
Running /usr/bin/time rustic ... or /usr/bin/time restic ... should suffice.
Even better would be to get a memory profile for your rustic check run...
Rustic's memory usage seems quite reasonable. This issue can be closed.
It appears that earlier OOM during check was a side effect of rustic caching in my test env. Without caching both restic and rustic finish check --no-cache in comparable amount of time and without issues.
restic
[...]
15478 additional files were found in the repo, which likely contain duplicate data.
You can run `restic prune` to correct this.
check snapshots, trees and blobs
no errors were found1 / 4751 snapshots
[32:06] 100.00% 4751 / 4751 snapshots
12768.21user 1063.38system 45:45.97elapsed 503%CPU (0avgtext+0avgdata 25802328maxresident)k
rustic
[...]
checking snapshots and trees...
- reading snapshots...
[00:00:01] ████████████████████████████████████████ 4751/4751
- checking trees...
[00:43:49] ████████████████████████████████████████ 4751/4751
6748.48user 10709.71system 48:54.87elapsed 594%CPU (0avgtext+0avgdata 11435812maxresident)k
Thanks for your results! Now, memory usage is very fine. And I'm also very satisfied with user CPU. But system CPU is very high - especially compared with restic.
Moreover total CPU usage is quite a bit higher than restic which is not expected... I have to think about possible system bottlenecks.
So I leave this issues open :wink:
Here are my numbers from a ~200GB repo (local, without cache):
restic check:
85.71user 2.80system 0:47.30elapsed 187%CPU (0avgtext+0avgdata 239752maxresident)k
6328inputs+8outputs (39major+66427minor)pagefaults 0swaps
rustic check:
40.84user 5.12system 0:31.00elapsed 148%CPU (0avgtext+0avgdata 64980maxresident)k
3720inputs+0outputs (0major+371270minor)pagefaults 0swaps
So user CPU always seems to be about 50% of restic's user CPU. In my case rustic's system CPU is double the restic one, in case of your big repo, its 10 times. So this might be an effect which is even growing for bigger (in terms of trees) snapshots...
Could be because of memory allocations, though. In my case rustic gets 5 times the minor pagefaults restic gets. @borkd: Can you also specify information about page faults for your example?
I tested to use the jemalloc allocator. In my case this did speed up rustic a bit, but left system cpu time and minor pagefaults still high.
There is now #54 which can be used for tests.
Level of parallelism may be behind some of the differences in execution profiles.
IIRC, stock restic parallelism was hardcoded to values which yield decent performance for typical consumer grade systems found in the wild. Finding those knobs and fiddling with them proved fruitful to experiment and balance the speed and resource use for different applications.
I have not looked at what rustic tries to do in terms of greed/most effective use of available resources, but I'm strongly in favor of sane defaults which are user tunable.
If the check requires 20M calls to filesystem where a repository is stored to complete that is what it takes. Host may have CPU cores and memory available to spin up excess workers, but they won't help much after backend becomes saturated.
Closing this as there seems to be no performance problem ATM. If you encounter any similar or other performance topic, feel free to open a new issue !
The high CPU consumption issue with rustic check still persists. Compared to restic check, rustic uses 4x the memory and 2x the CPU, while the overall runtime is roughly the same:
restic:
sudo systemd-run --wait -P -t /usr/bin/time resticprofile check
453.78user 101.24system 1:08.28elapsed 812%CPU (0avgtext+0avgdata 744904maxresident)k
283323408inputs+456outputs (13major+848140minor)pagefaults 0swaps
Service runtime: 1min 9.673s
CPU time consumed: 9min 15.050s
Memory peak: 24.5G (swap: 0B)
IO Bytes: read 135.1G, written 76K
rustic:
sudo systemd-run --wait -P -t /usr/bin/time rustic -p $(pwd)/key -r /backup/restic-repo check --read-data
660.78user 355.27system 1:07.88elapsed 1496%CPU (0avgtext+0avgdata 2930124maxresident)k
283008904inputs+6918904outputs (9major+5899658minor)pagefaults 0swaps
Service runtime: 1min 7.903s
CPU time consumed: 16min 56.067s
Memory peak: 24.9G (swap: 0B)
IO Bytes: read 134.9G, written 533.8M
Potentially relevant information:
rustic repoinfo
| File type | Count | Total Size |
|-----------|-------|------------|
| Key | 3 | 1.3 KiB |
| Snapshot | 747 | 342.6 KiB |
| Index | 119 | 71.6 MiB |
| Pack | 2029 | 134.7 GiB |
| Total | 2898 | 134.8 GiB |
| Blob type | Count | Total Size | Total Size in Packs |
|-----------|---------|------------|---------------------|
| Tree | 486147 | 1.7 GiB | 433.4 MiB |
| Data | 1084392 | 203.3 GiB | 134.2 GiB |
| Total | 1570539 | 205.0 GiB | 134.6 GiB |
| Blob type | Pack Count | Minimum Size | Maximum Size |
|------------|------------|--------------|--------------|
| Tree packs | 97 | 528 B | 92.0 MiB |
| Data packs | 1932 | 4.7 KiB | 95.6 MiB |
@seiuneko Can you please give the exact calling parameter you use when when running restic (in resticprofile) and rustic? And the outputs of both runs?
The higher memory usage can be due to rustic reading all packs into memory before checking them - and for parallel checks this could be a couple of 100MiBs (as you are having 100MiB pack sizes).
Also it seems to me that the rustic run did create the cache files withing your run (was writing quite some ammount of data which should be roughly the cache size). Do you use the same cache? Or, if this is a repo on a local disc, can you re-try with no-cache?
@aawsome
Can you please give the exact calling parameter you use when when running restic (in resticprofile) and rustic? And the outputs of both runs?
rustic check
❯❯❯ sudo systemd-run --wait -P -t /usr/bin/time rustic -p $(pwd)/key -r /backup/restic-repo check --read-data
Running as unit: run-p123426-i4316218.service
Press ^] three times within 1s to disconnect TTY.
[INFO] using no config file, none of these exist: /root/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml
[INFO] repository local:/backup/restic-repo: password is correct.
[INFO] using cache at /root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec
[00:00:00] getting snapshots... ████████████████████████████████████████ 780/780
[00:00:00] checking Snapshot in cache... ████████████████████████████████████████ 360.04 KiB/360.04 KiB 65.84 MiB/s (ETA -)
[00:00:00] reading index... ████████████████████████████████████████ 359/359
[00:00:00] listing packs...
[00:00:00] cleaning up packs from cache...
[00:00:00] checking trees... ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1/780 [WARN] Error in cache backend reading Pack,679b90f3: Error: Failed to read at offset `32809279` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/67/679b90f319930b466614de183f5657251341d480efd3f028be6cfc13b9a9aaf1` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:00] checking trees... ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 4/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `1316195` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:00] checking trees... ███████░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 140/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `9573575` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `44865407` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:00] checking trees... █████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ 262/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `13011300` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `44980734` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:00] checking trees... █████████████░░░░░░░░░░░░░░░░░░░░░░░░░░░ 265/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `12348921` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:00] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 300/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `75807717` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `29046936` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:00] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 301/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `33349284` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `41049536` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `16985886` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `14074707` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `6199594` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `68391928` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `3097947` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `50290668` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `38813419` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `31494868` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `54718303` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `1385144` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:01] checking trees... ███████████████░░░░░░░░░░░░░░░░░░░░░░░░░ 302/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `69037536` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `26845821` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 333/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `75597220` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `5638817` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 333/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `52701642` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 333/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `40989891` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 334/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `29514840` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `64739670` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 334/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `72771727` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 336/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `85397228` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 340/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `47068729` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `69075163` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 340/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `38515633` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `33127787` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:02] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 348/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `40352403` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:03] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 349/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `44432628` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `47144820` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `23094023` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:03] checking trees... █████████████████░░░░░░░░░░░░░░░░░░░░░░░ 349/780 [WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `51439076` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,1efe06c5: Error: Failed to read at offset `64497878` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/1e/1efe06c58fae274bb550ee81040b77124cc2252e98b013e25bd68cca14659e02` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:03] checking trees... ███████████████████░░░░░░░░░░░░░░░░░░░░░ 384/780 [WARN] Error in cache backend reading Pack,455a7a23: Error: Failed to read at offset `446376` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/45/455a7a230b52aa7e402e7c004e6b782bc6a01135a5ddbd3d0ec04ee5860007fc` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,60f8229b: Error: Failed to read at offset `481361` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/60/60f8229b06929e709446dd23336ea288c65e83e37b60f14cc290366dce77d48b` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,60f8229b: Error: Failed to read at offset `486580` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/60/60f8229b06929e709446dd23336ea288c65e83e37b60f14cc290366dce77d48b` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,60f8229b: Error: Failed to read at offset `482675` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/60/60f8229b06929e709446dd23336ea288c65e83e37b60f14cc290366dce77d48b` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,60f8229b: Error: Failed to read at offset `491352` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/60/60f8229b06929e709446dd23336ea288c65e83e37b60f14cc290366dce77d48b` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,f138ff08: Error: Failed to read at offset `843685` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/f1/f138ff08a6fb5a4d91e149360c239bd7b88ee911d501c1c209b863840b9f2012` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,d3d38b65: Error: Failed to read at offset `320771` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/d3/d3d38b65ba4382d263043370ae242a07611010115e94922718b52bb5d85c9204` (kind: related to input/output operations): caused by: failed to fill whole buffer
[WARN] Error in cache backend reading Pack,53c5fc04: Error: Failed to read at offset `530648` from file at `/root/.cache/rustic/21e70603b6d6fd79750a4d21d0dfccf2576be5a94f84304cb8def8907f040bec/data/53/53c5fc045a1802b2c27ee26f295174f1f4f491ee4bf47b0acdd4b2c8d43d246b` (kind: related to input/output operations): caused by: failed to fill whole buffer
[00:00:08] checking trees... ████████████████████████████████████████ 780/780
[00:00:57] reading pack data... ████████████████████████████████████████ 135.21 GiB/135.21 GiB 2.34 GiB/s (ETA -)
689.12user 374.81system 1:08.02elapsed 1563%CPU (0avgtext+0avgdata 2966764maxresident)k
284112784inputs+9103600outputs (9major+3722513minor)pagefaults 0swaps
Finished with result: success
Main processes terminated with: code=exited, status=0/SUCCESS
Service runtime: 1min 8.045s
CPU time consumed: 17min 43.943s
Memory peak: 25.8G (swap: 0B)
IO Bytes: read 135.4G, written 672.3M
restic check
❯❯❯ sudo systemd-run --wait -P -t /usr/bin/time restic -p $(pwd)/key -r /backup/restic-repo check --option=local.connections=10 --read-data
Running as unit: run-p103715-i4296511.service
Press ^] three times within 1s to disconnect TTY.
using temporary cache in /tmp/restic-check-cache-1222012002
create exclusive lock for repository
repository 21e70603 opened (version 2, compression level auto)
created new cache in /tmp/restic-check-cache-1222012002
load indexes
[0:00] 100.00% 359 / 359 index files loaded
check all packs
check snapshots, trees and blobs
[0:03] 100.00% 780 / 780 snapshots
read all data
[0:53] 100.00% 2306 / 2306 packs
no errors were found
452.63user 102.72system 0:59.23elapsed 937%CPU (0avgtext+0avgdata 620888maxresident)k
284537184inputs+456outputs (17major+734960minor)pagefaults 0swaps
Finished with result: success
Main processes terminated with: code=exited, status=0/SUCCESS
Service runtime: 59.251s
CPU time consumed: 9min 15.366s
Memory peak: 25.7G (swap: 0B)
IO Bytes: read 135.6G, written 268K
Do you use the same cache?
No
can you re-try with no-cache?
rustic check --no-cache
❯❯❯ sudo systemd-run --wait -P -t /usr/bin/time rustic -p $(pwd)/key -r /backup/restic-repo check --read-data --no-cache
Running as unit: run-p107812-i4300607.service
Press ^] three times within 1s to disconnect TTY.
[INFO] using no config file, none of these exist: /root/.config/rustic/rustic.toml, /etc/rustic/rustic.toml, ./rustic.toml
[INFO] repository local:/backup/restic-repo: password is correct.
[INFO] using no cache
[00:00:00] getting snapshots... ████████████████████████████████████████ 780/780
[00:00:00] reading index... ████████████████████████████████████████ 359/359
[00:00:00] listing packs...
[00:00:06] checking trees... ████████████████████████████████████████ 780/780
[00:00:58] reading pack data... ████████████████████████████████████████ 135.21 GiB/135.21 GiB 2.33 GiB/s (ETA -)
675.73user 372.22system 1:05.70elapsed 1594%CPU (0avgtext+0avgdata 2820224maxresident)k
284114800inputs+0outputs (9major+5930630minor)pagefaults 0swaps
Finished with result: success
Main processes terminated with: code=exited, status=0/SUCCESS
Service runtime: 1min 5.728s
CPU time consumed: 17min 27.961s
Memory peak: 26.5G (swap: 0B)
IO Bytes: read 135.4G