TW
About borg check: usually I let it verify all archives (so I only give the repo spec), but as you have a quite big repo (and maybe many archives), I...
You could try sending SIGUSR1 and SIGUSR2 to the borg process. That outputs some info and a traceback (but, IIRC, just continues after outputting that).
Line 812 is borg writing content data while extracting a file. Maybe try sending SIGUSR2 multiple times to see whether you get different locations in the code.
SIGUSR1 / SIGINFO should also print something IF borg is currently in `extract_item`. Maybe this is also helpful: https://github.com/borgbackup/borg/issues/2419#issuecomment-345118768
@Frostregen yeah, extract should "work" after that (but that one file will have that internal damage, where borg replaced the corrupt chunk with an all-zero chunk). If you still could...
Ah, good. Then restore the file, do another backup with that file, then run borg check --repair again to heal the archives.
Reopening it for reviewing it later whether the experience with this issue can be improved.
@Frostregen you have a lot of RAM in these machines, but you also have a rather big repo, so maybe check whether you do run out of RAM on client...
Hmm, bit out of ideas here. Maybe one could try to reproduce this issue on a developer machine and check if there is anything unusual going on, like getting stuck...
I suspect there is a bug somewhere (could be in borg or elsewhere).