borg icon indicating copy to clipboard operation
borg copied to clipboard

borg2: Repository.ObjectNotFound: Object with key XX

Open henfri opened this issue 6 months ago • 3 comments

Hello,

I am testing borg2 and get this error:

Repository.ObjectNotFound: Object with key archives/f315176e8a4baa1458c7a27374924bd11c8af25c7b027f594460472f99b333d3.del not found in repository ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg.
Command '/usr/local/bin/borg2 compact --remote-path /usr/local/bin/borg2 --umask 77 --info --repo ssh://omv_borg....

borg2 2.0.0b17

As borg2 is in beta-testing, it may be valuable to try to identify the underlying issue. FYI: I am running the same backup-commands onto another target, which may help.

I suppose borg check would be the next step?

Greetings, Hendrik

henfri avatar Jun 16 '25 06:06 henfri

Please make sure you use the latest beta and add the version number to the top post.

It looks like it can't find a soft-deleted (*.del) archive, which is a bit strange, because before accessing these archives it just lists the directory to see what's there.

borg delete and borg prune do soft-deletion of archives. These archive objects have no contents, they are just 0 bytes. Only the filename contains a pointer (archive id) to the main archive metadata block. Because it was already soft-deleted anyway, this might be a rather cosmetic issue.

Did you run multiple borg in parallel?

ThomasWaldmann avatar Jun 16 '25 10:06 ThomasWaldmann

Hello,

I am not aware running multiple borg in parallel. I use borg2 2.0.0b17, which seems to be the latest beta.

borg check is running now.

Greetings, Hendrik

henfri avatar Jun 16 '25 10:06 henfri

If you have pruned/deleted archives, run borg compact now and then.

See the docs: either fast(er) or with stats, whatever you prefer.

ThomasWaldmann avatar Jun 16 '25 10:06 ThomasWaldmann

Hello,

so first I ran check:

root@homeserver:~# /usr/local/bin/borg2 check --remote-path /usr/local/bin/borg2 --umask 77 --info --repo ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-
b72f-cde6585f8d8e/Backups/borg
Enter passphrase for key ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg:


Starting full repository check
Starting from beginning.
Checkpointing at key data/01674fbf3def739f20159c789c929624b5da0c82b3281880aadde070069212a5.
Checkpointing at key data/02cf912113e7425e99bb16e737f31c5783250943dac2f418343d8d6e7c8d97c2.
Checkpointing at key data/044c5338c1a12eeecffd93b256d1b78326c6be0bef3846bf48446aa93ca9be0c.
Checkpointing at key data/05d6e824edaaf0cc88c70b568b890bca12911ea0d4c62fc435fb3565747c1ee4.
Starting archive consistency check...
Checkpointing at key data/077c36a27eb2c5a4be6de997b28e7fb09dad22c19c8e41bb1d548e3bb64b73e0.
Checkpointing at key data/0936d92f0ece3b220fc4011d7968a35f2bfc8acaf9735416337b35a9157cfa73.
Checkpointing at key data/0ab0409ec04a40f6bc52971e0e192fa5b8a1aed7b1015de89d91d5cf73459bb3.
Checkpointing at key data/0c3e4a972dfb5ea4ededde34cbf20ef3a6ca4acdce9b7685f20bf26277d6f2c3.
Checkpointing at key data/0df92b8319f85b9ceab94c2b0b42b63f3b742b15604d5b7499c3004b922df5de.
Checkpointing at key data/0fd67fbcd4c5edb22b85964af6a166c6a3da552eb2b13ec88dc22f6343453c6a.
Checkpointing at key data/11b8a6ea5893a509c776eb596e3fb082bcc63172d915c8928f76c2811565c166.
Checkpointing at key data/139bc3ce466d8274f52f7b829baf5960f51e4801ef4011d115012643fc9faedd.
Checkpointing at key data/158916fbaa4408198c48699edbf1b7aec6d18eab6704112c03d7dec83f295479.
Checkpointing at key data/177650596b7246e3472cc02b58ea5ff571efe124b4fb1171bd03d8161dd44bdb.
Checkpointing at key data/196bbaaec3cdc7f4c0fdcc5c5ad816b8f1afed71f196596c8d6b8a8b6888572e.
Checkpointing at key data/1b5fc08408a079d8fb93d71778fd2223982213c81f479309cf9002c74e17686e.
Checkpointing at key data/1d23ea04af2a84b38493070a39f3e4f97cba3bc3fb9f939cc4bc20ddb253cbaa.
...
Checkpointing at key data/e4480da23234d544d5750ab64333cef70ddc38a521def6b0e4e8e300e38d176f.
Checkpointing at key data/e69e1f81149ce04e963cd49d05228df45aef56e18b4d6f6ce58318566f1a0668.
Checkpointing at key data/e90335ed81852901dc761bb5f2e5b6bbfffbd67a19fa77a236a83240419263c4.
Checkpointing at key data/eb7167df8cb4cb4b2f63186e9d9abfa3c825af43ebf831cff4b846db205d7e9d.
Checkpointing at key data/edd83117598e0b5d3c3e68b346db178010bab1209ed59a5017cd4df72899ac32.
Checkpointing at key data/f03a5feda9f3e7cdb7af65764a616798d5cddb031d6c64ca22a547c4e00beb4f.
Checkpointing at key data/f29fde27a4d3206871825207acc2d1bb549e7526f181550b12731e9ceb00290c.
Checkpointing at key data/f508073919e6a84e53994f6d6da36cf1ca71fa993c3b2a2a1a3333a1873929f5.
Checkpointing at key data/f71b4c74aceb91f454278e0889f12b9bc2dc72378a8582f08a5b47d55c61bc9e.
Checkpointing at key data/f94d719af8cc66478f4f058c4c7cb0d8600f041bad6684a1317eb285f9bd10f7.
Checkpointing at key data/fba63c61fd8ca2e0c66719c5782b6e5730f0fc79501dd83e2a6eb7efabe15cc4.
Checkpointing at key data/fe110ce6d9962e281fc336921ed7482d3c1c26652da42250fe139e78402dcc45.
Finished repository check.
Checked 1422995 repository objects, 0 errors.
Finished full repository check, no problems found.




Analyzing archive homeserver-documents-2025-03-20T20:55:44 2025-03-20 20:55:45.976636+01:00 f315176e8a4baa1458c7a27374924bd11c8af25c7b027f594460472f99b333d3 (1/33)
Analyzing archive homeserver-documents-2025-03-23T13:43:04 2025-03-23 13:43:05.515519+01:00 5f6ee823bf0212a46d6bcdd5db2b891365b77accc5a382349e26dd292d118a44 (2/33)
Analyzing archive homeserver-documents-2025-03-30T15:47:43 2025-03-30 15:47:45.349444+02:00 fa64346d051a704af30c56904c0ca08bde05cf04579fb0e57722c9ba4a1d984f (3/33)
Analyzing archive homeserver-documents-2025-03-31T12:03:03 2025-03-31 12:03:05.127292+02:00 42a6c167644713a7538107c9b5736fa83ca999499137c8ec69ef3f9d406831ff (4/33)
Analyzing archive homeserver-documents-2025-04-06T12:03:04 2025-04-06 12:03:05.791834+02:00 7975856a3d0ac4777b020bc876b8c104b10333b503237abfac13bfd6530c12c5 (5/33)
Analyzing archive homeserver-documents-2025-04-13T12:03:03 2025-04-13 12:03:05.653169+02:00 b91455eaa4cbb71697769c7f7eb319861112aeb0ebe0683e1dbb61ae4507db54 (6/33)
Analyzing archive homeserver-documents-2025-04-20T12:03:04 2025-04-20 12:03:05.767717+02:00 12dac4a8b5cd6dd9d4ef899a9258eaf76f2991fb46b80b58bbd0917d5c5f94b9 (7/33)
Analyzing archive homeserver-documents-2025-04-27T12:03:03 2025-04-27 12:03:05.635036+02:00 8cf25e100ddbf5e965aa518ef4462d67a40eb939a64aa8038bc3a4d7fd8ed925 (8/33)
Analyzing archive homeserver-documents-2025-04-30T12:03:03 2025-04-30 12:03:05.181354+02:00 a641b33d9a4b34ce8af9464ed1f61f6b3b68b28f0841a8fbf8efb1dd5a809030 (9/33)
Analyzing archive homeserver-documents-2025-05-04T12:03:03 2025-05-04 12:03:05.192933+02:00 4306b748b81cdc35dd89ddaa1298e69121c607e93f3907d24c31b416e633be49 (10/33)
Analyzing archive homeserver-documents-2025-05-11T12:03:03 2025-05-11 12:03:05.109598+02:00 3b93efe5e74b5337a9f43d84f1426f88d6b0a4fb676dd45758045c2e5a1bb2ee (11/33)
Analyzing archive homeserver-documents-2025-05-18T12:03:03 2025-05-18 12:03:05.373641+02:00 ccb49e215dd4b913750610519644e2f740c749612285302ee2528273e0df0b89 (12/33)
Analyzing archive homeserver-documents-2025-05-25T12:03:04 2025-05-25 12:03:05.708909+02:00 9938bc361a78a3fa26749788040c353db3d4117d66d84eb55f99b45cf98874f1 (13/33)
Analyzing archive homeserver-documents-2025-05-29T12:03:03 2025-05-29 12:03:05.149380+02:00 26e12483eef6aaff77aed714dcdfd332dac7eb8314278944cdf9c89cd9af6350 (14/33)
Analyzing archive homeserver-documents-2025-05-30T12:03:04 2025-05-30 12:03:06.541779+02:00 c3f9def4b5b59a06de70ee80e48f802e14e55736138fa04c2d8cadd53de87f10 (15/33)
Analyzing archive homeserver-documents-2025-05-31T12:03:04 2025-05-31 12:03:06.733239+02:00 fb66dfb023a5f330b340cf89f4e28dd5c83f89d6d9b88ea5d253fe4b02754430 (16/33)
Analyzing archive homeserver-documents-2025-06-01T12:03:04 2025-06-01 12:03:06.630087+02:00 50daf0201aab729b62149e3b80ef002c7976406d17fe6c379137f3a5e99ac561 (17/33)
Analyzing archive homeserver-documents-2025-06-02T21:57:55 2025-06-02 21:57:57.213037+02:00 2b351849392151a76a8dcb9384f05a275c5e0f9ad927d082ba898840e2d370ba (18/33)
Analyzing archive homeserver-documents-2025-06-03T23:11:17 2025-06-03 23:11:19.534998+02:00 913f2e9799e5dd8d0c61e3dc3d61fc5f122fb8aef45c2af3d70b8a00c4ebd984 (19/33)
Analyzing archive homeserver-documents-2025-06-04T17:22:43 2025-06-04 17:22:45.171959+02:00 ea463d6e957788478516b569790712f85ce988b2b6a8c45e2519a0fcc622530b (20/33)
Analyzing archive homeserver-documents-2025-06-05T12:35:27 2025-06-05 12:35:29.149669+02:00 03854a5c3c6e7725e81e27631b27594c484203bd5c19cd12a165f84a1eab711f (21/33)
Analyzing archive homeserver-documents-2025-06-06T12:03:12 2025-06-06 12:03:19.039722+02:00 aae9c98fe3ec98ec4f5e0a80518243a1731afab6c4069b7cd2d8ce97275d70f6 (22/33)
Analyzing archive homeserver-documents-2025-06-07T17:25:18 2025-06-07 17:25:19.973157+02:00 3370a04aa364584615f33155cae5a17685c0970ddc5dc096ec818ba6ae253ebb (23/33)
Analyzing archive homeserver-documents-2025-06-08T12:35:17 2025-06-08 12:35:19.087210+02:00 1273f6df63880b64d8c398f6c537f7200f30c05cd2e9a0ae290dbdd83b67d77f (24/33)
Analyzing archive homeserver-documents-2025-06-09T12:35:07 2025-06-09 12:35:09.665963+02:00 503a35c7ae9e30b01f1182b21c166a2ba7cb3ef90e3ed6bbde751a02e4f24cba (25/33)
Analyzing archive homeserver-documents-2025-06-10T12:35:58 2025-06-10 12:36:00.133439+02:00 5b3012e03a87392c5f19252aaa9fab065a4cdb6e3beb7a088cf77466b3cd8960 (26/33)
Analyzing archive homeserver-documents-2025-06-11T12:35:50 2025-06-11 12:35:52.478136+02:00 db4ec1b10a0106054170965943c076b5ac9b75d96005fefa432998334e6b9858 (27/33)
Analyzing archive homeserver-documents-2025-06-12T12:35:51 2025-06-12 12:35:53.126630+02:00 885dbc35db3aa39170ecdc5c9c566ffe73082ca431c37098ef4965581c09ddf0 (28/33)
Analyzing archive homeserver-documents-2025-06-13T05:29:05 2025-06-13 05:29:07.179484+02:00 7027cee864b86d9c1d2893d944a0ab077a45125492cf1cf5b18409cb4ec6389c (29/33)
Analyzing archive homeserver-documents-2025-06-13T12:35:33 2025-06-13 12:35:35.869524+02:00 5c293c221e41df4f2306ad6498a6206eeccd55ca15b30482d1bca885daea1d61 (30/33)
Analyzing archive homeserver-documents-2025-06-14T12:35:59 2025-06-14 12:36:01.070429+02:00 4f5408260738fe4ad4fc6081903c2b04c2a3cb721703ff6552276ee2edb609e7 (31/33)
Analyzing archive homeserver-documents-2025-06-15T12:35:44 2025-06-15 12:35:46.054929+02:00 b45aa4c6b2745cf170f914ea841525f62dfe1d87cd67b2bbdfc6cb6cbb72944c (32/33)
Analyzing archive homeserver-documents-2025-06-15T23:35:20 2025-06-15 23:35:21.870309+02:00 ee6bf8941e6b77c3c8175f6869089ed2a5e6fddd20d9fd8b1b9e1f5b1b79a74a (33/33)
Archive consistency check complete, no problems found.

then compact:


root@homeserver:~# /usr/local/bin/borg2 compact --remote-path /usr/local/bin/borg2 --umask 77 --info --repo ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c
8-b72f-cde6585f8d8e/Backups/borg
Enter passphrase for key ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg:
Starting compaction / garbage collection...
Getting object IDs from cached chunks index...
Computing object IDs used by archives...
Analyzing archive homeserver-documents-2025-03-20T20:55:44 2025-03-20 20:55:45.976636+01:00 f315176e8a4baa1458c7a27374924bd11c8af25c7b027f594460472f99b333d3 (1/33)
Analyzing archive homeserver-documents-2025-03-23T13:43:04 2025-03-23 13:43:05.515519+01:00 5f6ee823bf0212a46d6bcdd5db2b891365b77accc5a382349e26dd292d118a44 (2/33)
Analyzing archive homeserver-documents-2025-03-30T15:47:43 2025-03-30 15:47:45.349444+02:00 fa64346d051a704af30c56904c0ca08bde05cf04579fb0e57722c9ba4a1d984f (3/33)
Analyzing archive homeserver-documents-2025-03-31T12:03:03 2025-03-31 12:03:05.127292+02:00 42a6c167644713a7538107c9b5736fa83ca999499137c8ec69ef3f9d406831ff (4/33)
Analyzing archive homeserver-documents-2025-04-06T12:03:04 2025-04-06 12:03:05.791834+02:00 7975856a3d0ac4777b020bc876b8c104b10333b503237abfac13bfd6530c12c5 (5/33)
Analyzing archive homeserver-documents-2025-04-13T12:03:03 2025-04-13 12:03:05.653169+02:00 b91455eaa4cbb71697769c7f7eb319861112aeb0ebe0683e1dbb61ae4507db54 (6/33)
Analyzing archive homeserver-documents-2025-04-20T12:03:04 2025-04-20 12:03:05.767717+02:00 12dac4a8b5cd6dd9d4ef899a9258eaf76f2991fb46b80b58bbd0917d5c5f94b9 (7/33)
Analyzing archive homeserver-documents-2025-04-27T12:03:03 2025-04-27 12:03:05.635036+02:00 8cf25e100ddbf5e965aa518ef4462d67a40eb939a64aa8038bc3a4d7fd8ed925 (8/33)
Analyzing archive homeserver-documents-2025-04-30T12:03:03 2025-04-30 12:03:05.181354+02:00 a641b33d9a4b34ce8af9464ed1f61f6b3b68b28f0841a8fbf8efb1dd5a809030 (9/33)
Analyzing archive homeserver-documents-2025-05-04T12:03:03 2025-05-04 12:03:05.192933+02:00 4306b748b81cdc35dd89ddaa1298e69121c607e93f3907d24c31b416e633be49 (10/33)
Analyzing archive homeserver-documents-2025-05-11T12:03:03 2025-05-11 12:03:05.109598+02:00 3b93efe5e74b5337a9f43d84f1426f88d6b0a4fb676dd45758045c2e5a1bb2ee (11/33)
Analyzing archive homeserver-documents-2025-05-18T12:03:03 2025-05-18 12:03:05.373641+02:00 ccb49e215dd4b913750610519644e2f740c749612285302ee2528273e0df0b89 (12/33)
...
Analyzing archive homeserver-documents-2025-06-01T12:03:04 2025-06-01 12:03:06.630087+02:00 50daf0201aab729b62149e3b80ef002c7976406d17fe6c379137f3a5e99ac561 (17/33)
Analyzing archive homeserver-documents-2025-06-02T21:57:55 2025-06-02 21:57:57.213037+02:00 2b351849392151a76a8dcb9384f05a275c5e0f9ad927d082ba898840e2d370ba (18/33)
Analyzing archive homeserver-documents-2025-06-03T23:11:17 2025-06-03 23:11:19.534998+02:00 913f2e9799e5dd8d0c61e3dc3d61fc5f122fb8aef45c2af3d70b8a00c4ebd984 (19/33)
Analyzing archive homeserver-documents-2025-06-04T17:22:43 2025-06-04 17:22:45.171959+02:00 ea463d6e957788478516b569790712f85ce988b2b6a8c45e2519a0fcc622530b (20/33)
Analyzing archive homeserver-documents-2025-06-05T12:35:27 2025-06-05 12:35:29.149669+02:00 03854a5c3c6e7725e81e27631b27594c484203bd5c19cd12a165f84a1eab711f (21/33)
Analyzing archive homeserver-documents-2025-06-06T12:03:12 2025-06-06 12:03:19.039722+02:00 aae9c98fe3ec98ec4f5e0a80518243a1731afab6c4069b7cd2d8ce97275d70f6 (22/33)
Analyzing archive homeserver-documents-2025-06-07T17:25:18 2025-06-07 17:25:19.973157+02:00 3370a04aa364584615f33155cae5a17685c0970ddc5dc096ec818ba6ae253ebb (23/33)
Analyzing archive homeserver-documents-2025-06-08T12:35:17 2025-06-08 12:35:19.087210+02:00 1273f6df63880b64d8c398f6c537f7200f30c05cd2e9a0ae290dbdd83b67d77f (24/33)
Analyzing archive homeserver-documents-2025-06-09T12:35:07 2025-06-09 12:35:09.665963+02:00 503a35c7ae9e30b01f1182b21c166a2ba7cb3ef90e3ed6bbde751a02e4f24cba (25/33)
Analyzing archive homeserver-documents-2025-06-10T12:35:58 2025-06-10 12:36:00.133439+02:00 5b3012e03a87392c5f19252aaa9fab065a4cdb6e3beb7a088cf77466b3cd8960 (26/33)
Analyzing archive homeserver-documents-2025-06-11T12:35:50 2025-06-11 12:35:52.478136+02:00 db4ec1b10a0106054170965943c076b5ac9b75d96005fefa432998334e6b9858 (27/33)
Analyzing archive homeserver-documents-2025-06-12T12:35:51 2025-06-12 12:35:53.126630+02:00 885dbc35db3aa39170ecdc5c9c566ffe73082ca431c37098ef4965581c09ddf0 (28/33)
Analyzing archive homeserver-documents-2025-06-13T05:29:05 2025-06-13 05:29:07.179484+02:00 7027cee864b86d9c1d2893d944a0ab077a45125492cf1cf5b18409cb4ec6389c (29/33)
Analyzing archive homeserver-documents-2025-06-13T12:35:33 2025-06-13 12:35:35.869524+02:00 5c293c221e41df4f2306ad6498a6206eeccd55ca15b30482d1bca885daea1d61 (30/33)
Analyzing archive homeserver-documents-2025-06-14T12:35:59 2025-06-14 12:36:01.070429+02:00 4f5408260738fe4ad4fc6081903c2b04c2a3cb721703ff6552276ee2edb609e7 (31/33)
Analyzing archive homeserver-documents-2025-06-15T12:35:44 2025-06-15 12:35:46.054929+02:00 b45aa4c6b2745cf170f914ea841525f62dfe1d87cd67b2bbdfc6cb6cbb72944c (32/33)
Analyzing archive homeserver-documents-2025-06-15T23:35:20 2025-06-15 23:35:21.870309+02:00 ee6bf8941e6b77c3c8175f6869089ed2a5e6fddd20d9fd8b1b9e1f5b1b79a74a (33/33)
Cleaning archives directory from soft-deleted archives...
Object with key archives/f315176e8a4baa1458c7a27374924bd11c8af25c7b027f594460472f99b333d3.del not found in repository ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg.

Error:

Repository.ObjectNotFound: Object with key archives/f315176e8a4baa1458c7a27374924bd11c8af25c7b027f594460472f99b333d3.del not found in repository ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg.

If reporting bugs, please include the following:

Traceback (most recent call last):
  File "borg/archiver/__init__.py", line 647, in main
  File "borg/archiver/__init__.py", line 541, in run
  File "borg/archiver/_common.py", line 143, in wrapper
  File "borg/archiver/compact_cmd.py", line 213, in do_compact
  File "borg/archiver/compact_cmd.py", line 46, in garbage_collect
  File "borg/archiver/compact_cmd.py", line 168, in report_and_delete
  File "borg/manifest.py", line 351, in nuke_by_id
  File "borg/remote.py", line 500, in do_rpc
  File "borg/remote.py", line 720, in call
  File "borg/remote.py", line 819, in call_many
  File "borg/remote.py", line 778, in handle_error
borg.repository.Repository.ObjectNotFound: Object with key archives/f315176e8a4baa1458c7a27374924bd11c8af25c7b027f594460472f99b333d3.del not found in repository ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg.

Platform: Linux homeserver 6.12.22+bpo-rt-amd64 #1 SMP PREEMPT_RT Debian 6.12.22-1~bpo12+1 (2025-04-25) x86_64
Linux: Unknown Linux
Borg: 2.0.0b17  Python: CPython 3.12.10 msgpack: 1.1.0 fuse: llfuse 1.5.1 [pyfuse3,llfuse]
PID: 1442535  CWD: /root
sys.argv: ['/usr/local/bin/borg2', 'compact', '--remote-path', '/usr/local/bin/borg2', '--umask', '77', '--info', '--repo', 'ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg']
SSH_ORIGINAL_COMMAND: None

does that help?

Greetings, Hendrik

henfri avatar Jun 16 '25 20:06 henfri

Can you run this on the client and on the server and check if both use 2.0.0b17:

/usr/local/bin/borg2 -V

ThomasWaldmann avatar Jun 16 '25 20:06 ThomasWaldmann

Yes:

henfri@homeserveralt:~$ /usr/local/bin/borg2 -V
borg2 2.0.0b17
root@homeserver:~# /usr/local/bin/borg2 -V
borg2 2.0.0b17

Gruß, Hendrik

henfri avatar Jun 16 '25 21:06 henfri

Hmm, strange. Please run this on the server:

ls -l REPODIR/archives   # if it is a lot, add | grep del

ThomasWaldmann avatar Jun 16 '25 21:06 ThomasWaldmann

My apologies. as mentioned, I am running this backup with borgmatic to two targets. And indeed, the source and one of the destinations (homeserveralt, see above) have b17. But the omv_borg_wireguard one has still b14.

root@openmediavault:/srv# ls -l /srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg/archives  |grep .del
-rw------- 1 borg     users 0 29. Apr 18:03 015f9cd8697ffd3279bdbfb1f37af977b322b203b3566a89982c227971c01c4e.del
-rw------- 1 borg     users 0 11. Apr 17:48 032ca1d40402131cc7c28c8fe77514002f8a10f939f62561f8516f9f6524a80b.del
-rw------- 1 borg     users 0 24. Apr 17:55 03e0008b5926fe5490dff58a11fe8b6c31097101f7ab3591e5ecd18dae7b1ab4.del
-rw------- 1 borg     users 0 14. Mai 17:50 0ad1e04babc46451406a08530f7bca2b645003835cfc1ae7359b1173d3d2f54b.del
-rw------- 1 borg     users 0 15. Apr 19:21 0ee77d5219fa991d92e978baa44c1adfa285e6a87b7fdb44c0d9507b01794fd6.del
-rw------- 1 borg     users 0 10. Mai 18:00 1626490e441e63eb0fe7c3b628b12bb34a59e0082ac09a147e39a5765bba1e68.del
-rw------- 1 borg     users 0  6. Mai 17:53 1dd3b4dea1ed3acb7665a808f1a3d27523174b8188ea5a6e0ea4f67bdec972eb.del
-rw------- 1 borg     users 0 12. Apr 17:49 1ebc143b7bddbe4ac96725d74e7b44ca06b6d1f7af075daaad02e392c8e64cfb.del
-rw------- 1 borg     users 0 21. Apr 18:03 1f05163d015f721ef477e795342e5bfdfe8e330f9f33de3bbeeb7efdb4067f49.del
-rw------- 1 borg     users 0 22. Mär 16:02 218ca305b9c52d99c4205d264c85938dc26731e1a7b011d11fcce27cb2e6a269.del
-rw------- 1 borg     users 0  8. Apr 18:00 227be0910e838f957fa8df10985e56b06898103aebab5a849bb9a4d673907b3d.del
-rw------- 1 borg     users 0 16. Apr 18:15 24a03841b5d25be4d2ffea73099fee5ade0c359eaf40dffb6798a64815ee2923.del
-rw------- 1 borg     users 0  5. Mai 17:54 2a1020d7dfcd0041a60a45b1a64f0fc0724e500ec59ea0ab4dbe28d5788f5c04.del
-rw------- 1 borg     users 0 14. Apr 18:02 2a26f8ccb63056c5b863d0351dfc5c6443478349133722087ab55eb99cd74be1.del
-rw------- 1 borg     users 0 19. Apr 21:24 3696a725072db0f57a868b53fc1c5f122dfb05be2584d77bbe15308ca70f0ff5.del
-rw------- 1 borg     users 0  1. Apr 18:01 36e560fe339b3b855541f618f245f736080b2730c2bfd677e9b6da8285e801c6.del
-rw------- 1 borg     users 0  7. Mai 17:49 3ea94969bc7722ab7dc5c1974d9840d46d829f54743847a720572714a0659d02.del
-rw------- 1 borg     users 0 17. Mai 18:15 41df41b63e4f62d85f8889a8e17e3676f47724c61ac111bde158c23b745e4ff9.del
-rw------- 1 borg     users 0  9. Apr 17:49 428a8e0c654128e223ee6d63ba74ccc7c68539b4825bece089b828f5169028c7.del
-rw------- 1 borg     users 0  3. Jun 13:13 429639268e59bc48a23b79097d3d994305dc9c7a3ecc3d10ed8f3e21f30ea40d.del
-rw------- 1 borg     users 0 17. Apr 22:39 452d970a7afdeada4f75b03b11cd53397ac77a1d7b19a33fc04260d13d12b126.del
-rw------- 1 borg     users 0 16. Mai 18:05 475c12c05f1c8a0734b3893c7f2877a3156856532f81d973051210601dbd0b60.del
-rw------- 1 borg     users 0 15. Mai 18:01 4ce6f9cb6122224ffb1d57cc6aba9f4e745b7ee0e8f6a19c5cbbc9ba9fa5bb08.del
-rw------- 1 borg     users 0 23. Apr 00:23 4dcd94384c5a68151b8f18656a3c8961b0ab997ddf770bda1e6d347ddd8d5eb3.del
-rw------- 1 borg     users 0 28. Mai 18:00 51c7cd527fdb79800e7a734161aa3a1314f99bf53ac8e997d439b816dc80052b.del
-rw------- 1 borg     users 0 12. Mai 17:49 52a73ec9b4f0cd5a373dc846c91a0c59ecdcddb1e060c0d8f81de434c46397a0.del
-rw------- 1 borg     users 0 23. Mai 18:44 5745a2e573ea6a9aeffe11bc9889fd067f7b588056711e105ae515cddb1e753b.del
-rw------- 1 borg     users 0  4. Apr 17:50 5bae6a83525033f4d565be35530454756e5b48a574d4082a495dcfaa5b573ea3.del
-rw------- 1 borg     users 0  3. Jun 17:28 5bb17092c79c56c27c2524c32ff44a5208677ed6450df7bfcb9e80bf7f85227e.del
-rw------- 1 borg     users 0  1. Mai 17:44 6ad05604f5f3b06659e1bd62bbf1abb765acc57ee22d5eff79a103ae73d70337.del
-rw------- 1 borg     users 0  2. Mai 17:49 71167ef060b1665f4f249a80108f894826b300211069fbbbf4ebf0e7787363fb.del
-rw------- 1 borg     users 0  3. Apr 18:03 78e293c87aeb9b5274dcab6efa2215f122b198f443cfafa74204082bbbc51564.del
-rw------- 1 borg     users 0 28. Apr 18:06 824b80cc0c1c778f04cc4580e3d33d97eb55292fa8e03094a1043ff9f900b35f.del
-rw------- 1 borg     users 0  8. Jun 05:50 842e4f6a6a537c44b04c268c4eb035262f03b3ca10c3126cc493977e8f07f8dc.del
-rw------- 1 borg     users 0 18. Apr 17:50 8f72948c255d5a375aa043b716be01bcc246549844d3ec0930cf3b189d355fe7.del
-rw------- 1 borg     users 0  3. Mai 17:55 95334e7a7a1656a275c2026ba13712966569f0f28402a90d0477dedb6a5c4f75.del
-rw------- 1 borg     users 0  8. Mai 17:56 95f0f71b9bb47ab25cdecad863f782c27d3470a91320aceb2a0fd37b9919681e.del
-rw------- 1 borg     users 0 23. Apr 17:50 97d67f10329fd17754ceab2c10a31df506251ff4f0fd9d67098535baba6c3b2b.del
-rw------- 1 borg     users 0 27. Mai 17:54 a509e5f4ff0968040b38b35b9d0adf44eba20980302b7cde54cce5831eac6618.del
-rw------- 1 borg     users 0  7. Apr 17:49 b144613c7dbd9cba1bdcd6de2ec3c61995a8471aa418d788e03833c3d642b37d.del
-rw------- 1 borg     users 0 21. Mär 16:29 b21ff5d59154068c3ca5d538ce2ec56e1e6df6e446270a7b3e26ac9dc3908740.del
-rw------- 1 borg     users 0  9. Mai 17:51 b2a2504d82dfb5c43026a413847f070e15af2dc550fe538cc2b4251b11798120.del
-rw------- 1 borg     users 0 17. Apr 20:52 bb8f7ae25b26dbae0a3e06dc8317a42b06c4e46475d031365e61bf4bf62e742c.del
-rw------- 1 borg     users 0 10. Apr 18:01 be42492c030fa5cdd2f1ad69f96abf0111ef02d216306366d0c27e306e1faf4b.del
-rw------- 1 borg     users 0 25. Apr 18:01 c5b972048af86db1d47ba73463e4754f27908ba3b502089d26c575c2df0b4dc6.del
-rw------- 1 borg     users 0  3. Jun 23:38 c6ed87ca6bfacb5bada1c48a5d42ad12b9d326c84a8e7015cf65e9dd8506f53a.del
-rw------- 1 borg     users 0 13. Mai 17:58 c85662881fe82ad3202bdc96b36b853e81fd24d12c8811ffb9afb59beb15b1dd.del
-rw------- 1 borg     users 0  2. Jun 21:19 cadccce9e7a0d9fa2a7b77159951839dbfd2975a6446160669a40fe4356bb1c6.del
-rw------- 1 borg     users 0 19. Apr 17:56 cd489d02edf0266b1643a8d47919e8e03e3017b85653e633b35b9bea9a354f43.del
-rw------- 1 borg     users 0  7. Jun 12:40 cf15efc4cfb307d3bc1db4f8efd5b7e9c6bbd7c46385c20bee59788028cc7a40.del
-rw------- 1 borg     users 0 20. Mai 17:52 d15bf39759d48bf9188dc2b2abe842a643b23f3e4a816e6ed5e35d76601d18e7.del
-rw------- 1 borg     users 0 24. Mai 17:58 dce04007a4e9f5e34cdb25fa2ee1c8781593704bae11311b2a4c3eff9616f565.del
-rw------- 1 borg     users 0  3. Jun 08:53 df9dadf11b2c9a6206ec7b0a3fa1eea971777887b54e44733c1519eb40b3aa75.del
-rw------- 1 borg     users 0 23. Mai 07:01 e190cfec0262be61e087847d84b584ad1984b2162dd778f9ae46bed59664fcd3.del
-rw------- 1 borg     users 0 16. Apr 19:16 ed3a819bfdebd8286dc3eb21998f26f0fd3fdaefd8a1ba8b52a0d4b56723af44.del
-rw------- 1 borg     users 0 19. Mai 17:58 ee41ee6ff1c95ebffd5875bc352c9535a9722f03476efbbeb8d3ceaf150b4b17.del
-rw------- 1 borg     users 0  5. Apr 17:49 f5457daecfeef57f63d56cd1d8ad21e937033237be2ac09047dd8cb8a74794bd.del
-rw------- 1 borg     users 0  2. Apr 18:01 f8c9c739f727354bbd3039c0fd3a2f8e8cbe64183c66190ed517206f463a6ca9.del
-rw------- 1 borg     users 0 26. Apr 17:53 fc8491451479c98b70e6b52cdb8ef9f23bf05c98d965ec0aa89e343eae8be303.del
-rw------- 1 borg     users 0 26. Mai 17:49 febdb27f149561d7eaa14aa8041db090bb664a3f98d4d72571bbd634809f4371.del

However, the compact and check comands were run on the right machine

Repository.ObjectNotFound: Object with key archives/f315176e8a4baa1458c7a27374924bd11c8af25c7b027f594460472f99b333d3.del not found in repository ssh://omv_borg_wireguard//srv/dev-disk-by-uuid-7ce96b63-64f2-44c8-b72f-cde6585f8d8e/Backups/borg.
Command '/usr/local/bin/borg2 compact --remote-path /usr/local/bin/borg2 --umask 77 --info --repo ssh://omv_borg...

So "only" the information that I am running the same borg version on both machines is wrong.

Greetings, Hendrik

henfri avatar Jun 16 '25 21:06 henfri

There were incompatible changes, so please update all machines to same version.

The compact log which showed the issue was with the b14 server, so guess this can be the root cause.

ThomasWaldmann avatar Jun 17 '25 09:06 ThomasWaldmann

Ok, thanks.

Is it possible to repair or shall I recreate?

henfri avatar Jun 17 '25 09:06 henfri

Not sure, maybe just try running compact again.

If you see further issues in future in that repo, kill it and create a new one.

ThomasWaldmann avatar Jun 17 '25 10:06 ThomasWaldmann

Closing this for now. In case you find the same issue again with b17/b17, please reopen.

ThomasWaldmann avatar Jun 17 '25 10:06 ThomasWaldmann

Hello,

the update of the server to b17 fixed the issue. Check and Compact were successful.

Greetings, Hendrik

henfri avatar Jun 20 '25 07:06 henfri