jj icon indicating copy to clipboard operation
jj copied to clipboard
Published 1 month ago • verified publisher icon jj-vcs

hosed jj repo after hard-reboot.

Open hanwen opened this issue 1 year ago • 17 comments

Description

I had to reboot my machine, and shutdown refused to work, so I power cycled my machine (I didn't run sync before). After restarting, jj refuses to work in my repo.

I get:

$ jj-0.21 log
Internal error: Failed to load the repo
Caused by:
1: Error when reading object  of type view
2: Is a directory (os error 21)
hanwen@fedora:~/vc/go-fuse$

this comes from

open("/home/hanwen/vc/go-fuse/.jj/repo/op_store/views/", O_RDONLY|O_LARGEFILE|O_CLOEXEC) = 5
fcntl(5, F_SETFD, FD_CLOEXEC)           = 0
fstat(5, {st_mode=S_IFDIR|0755, st_size=221440, ...}) = 0
mmap(NULL, 221460, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f22c8876000
read(5, 0x7f22c8876150, 221440)         = -1 EISDIR (Is a directory)

I guess there is a filename in the views directory that should not be empty, but somehow ended up empty.

I am at a loss how to fix this, but I can probably recover by starting a new repository and patching my work in progress..

  • Platform: linux x86_64
  • Version: I was using jj 0.7 when I did the reboot, other versions exhibit the same problem.

hanwen avatar Sep 08 '24 20:09 hanwen

Assuming it's only some recent writes that failed, you should be able to restore it manually by doing this:

  1. Run jj --ignore-working-copy op log -T id. Hopefully this command succeeds
  2. Going back one operation at a time, delete .jj/repo/op_heads/heads/<current operation id> and touch .jj/op_heads/heads/<previous operation id> (only the name in hex matters, the file can be empty)

martinvonz avatar Sep 08 '24 21:09 martinvonz

it looks like there is only one operation?! Setting the head to 000000.. didn't work.

$ jj-0.21 --ignore-working-copy  op log -T id  | cat
@  08dfc82b83f0cb2b190d8d3064543a290670be312760c58f480299e59c0f8aaec9f00795a6d70eb40a82566a0cce997474449cddfa45ffffe662a227c7092a3f
○  00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
hanwen@fedora:~/vc/go-fuse$ tar cf ../gofuse-hosed.tar .jj
hanwen@fedora:~/vc/go-fuse$ rm .jj/repo/op_heads/heads/08dfc82b83f0cb2b190d8d3064543a290670be312760c58f480299e59c0f8aaec9f00795a6d70eb40a82566a0cce997474449cddfa45ffffe662a227c7092a3f 
hanwen@fedora:~/vc/go-fuse$ jj log
Internal error: Corrupt repository: there are no operations
hanwen@fedora:~/vc/go-fuse$ touch .jj/repo/op_heads/heads/00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000
hanwen@fedora:~/vc/go-fuse$ jj log
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: NotFound', /home/runner/work/jj/jj/lib/src/op_heads_store.rs:90:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

hanwen avatar Sep 09 '24 05:09 hanwen

Try some recent files found in .jj/repo/op_store/operations

https://github.com/martinvonz/jj/issues/3370

yuja avatar Sep 09 '24 06:09 yuja

no go, but it fails with different error messages:

ls -ltra .jj/repo/op_store/operations

...
-rw-------. 1 hanwen hanwen    248 Sep  8 17:04 1eb13476fb32c892ab37ff3ce313d3ebb37d859821245a5e705c629914325072aeec8a192f65cffcfdb03361045200bcf842039f233b82ce055c6e7dfb7ffc09
-rw-------. 1 hanwen hanwen    195 Sep  8 17:05 b458a71f3b6fe0a8d16b96a2fea499f70cd0389936d4fec1ee4e64fb086d5e78a7d103c0db2bea885b44a6d85bc5784ea0753a5ae8f6e38b3fea60f84252f1b1
-rw-------. 1 hanwen hanwen    195 Sep  8 17:17 54d76eb5f6a20da79c32091d4a1ebb0e0e079af6f6e12fb060cd93d5024116a0a92753b8583f3c74d5c345f09c9be65d1d5e60e3191ac8b61b922f5833b90f5a
-rw-------. 1 hanwen hanwen    195 Sep  8 17:43 3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464
-rw-------. 1 hanwen hanwen    195 Sep  8 21:53 96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33
-rw-------. 1 hanwen hanwen    195 Sep  8 21:55 58c3bc3179facd5d14211c62ee11182a4ea14ec2bca6ae8a3f990cfa90b1300a995b3def1e9a305e22ee8b38477ec9ec3e76368986466caf58c14f3c5d6b094c
-rw-------. 1 hanwen hanwen    195 Sep  8 22:03 d291b16b2b2cdec761c961976542eedaeafc9cb537383312a5bec20742a99cae3148ca36ced4fe03eda64f9276989e88d17d62c68035da8bbe8ce5ef1cd68c67
drwxr-xr-x. 1 hanwen hanwen     38 Sep  8 22:03 ..
-rw-r--r--. 1 hanwen hanwen      0 Sep  8 22:31 08dfc82b83f0cb2b190d8d3064543a290670be312760c58f480299e59c0f8aaec9f00795a6d70eb40a82566a0cce997474449cddfa45ffffe662a227c7092a3f
drwxr-xr-x. 1 hanwen hanwen 222464 Sep  8 22:31 .
hanwen@fedora:~/vc/go-fuse$ touch .jj/repo/op_heads/heads/d291b16b2b2cdec761c961976542eedaeafc9cb537383312a5bec20742a99cae3148ca36ced4fe03eda64f9276989e88d17d62c68035da8bbe8ce5ef1cd68c67
hanwen@fedora:~/vc/go-fuse$ jj log
Concurrent modification detected, resolving automatically.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other("Is a directory (os error 21)")', lib/src/operation.rs:97:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
hanwen@fedora:~/vc/go-fuse$ cp .jj/repo/op_store/operations/d291b16b2b2cdec761c961976542eedaeafc9cb537383312a5bec20742a99cae3148ca36ced4fe03eda64f9276989e88d17d62c68035da8bbe8ce5ef1cd68c67  .jj/repo/op_heads/heads/
hanwen@fedora:~/vc/go-fuse$ jj log
Concurrent modification detected, resolving automatically.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other("Is a directory (os error 21)")', lib/src/operation.rs:97:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
hanwen@fedora:~/vc/go-fuse$ rm .jj/repo/op_heads/heads/d291b16b2b2cdec761c961976542eedaeafc9cb537383312a5bec20742a99cae3148ca36ced4fe03eda64f9276989e88d17d62c68035da8bbe8ce5ef1cd68c67; cp  .jj/repo/op_store/operations/58c3bc3179facd5d14211c62ee11182a4ea14ec2bca6ae8a3f990cfa90b1300a995b3def1e9a305e22ee8b38477ec9ec3e76368986466caf58c14f3c5d6b094c .jj/repo/op_heads/heads/
hanwen@fedora:~/vc/go-fuse$ jj log
Concurrent modification detected, resolving automatically.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other("Is a directory (os error 21)")', lib/src/operation.rs:97:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
hanwen@fedora:~/vc/go-fuse$ >.jj/repo/op_heads/heads/58c3bc3179facd5d14211c62ee11182a4ea14ec2bca6ae8a3f990cfa90b1300a995b3def1e9a305e22ee8b38477ec9ec3e76368986466caf58c14f3c5d6b094c
hanwen@fedora:~/vc/go-fuse$ jj log
Concurrent modification detected, resolving automatically.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other("Is a directory (os error 21)")', lib/src/operation.rs:97:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
hanwen@fedora:~/vc/go-fuse$ cp .jj/repo/op_store/operations/58c3bc3179facd5d14211c62ee11182a4ea14ec2bca6ae8a3f990cfa90b1300a995b3def1e9a305e22ee8b38477ec9ec3e76368986466caf58c14f3c5d6b094c  .jj/repo/op_heads/heads/
hanwen@fedora:~/vc/go-fuse$ jj log
Concurrent modification detected, resolving automatically.
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other("Is a directory (os error 21)")', lib/src/operation.rs:97:64
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
hanwen@fedora:~/vc/go-fuse$ cp .jj/repo/op_store/operations/96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33  .jj/repo/op_heads/heads/
hanwen@fedora:~/vc/go-fuse$ rm .jj/repo/op_heads/heads/
08dfc82b83f0cb2b190d8d3064543a290670be312760c58f480299e59c0f8aaec9f00795a6d70eb40a82566a0cce997474449cddfa45ffffe662a227c7092a3f
58c3bc3179facd5d14211c62ee11182a4ea14ec2bca6ae8a3f990cfa90b1300a995b3def1e9a305e22ee8b38477ec9ec3e76368986466caf58c14f3c5d6b094c
96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33
hanwen@fedora:~/vc/go-fuse$ rm .jj/repo/op_heads/heads/08dfc82b83f0cb2b190d8d3064543a290670be312760c58f480299e59c0f8aaec9f00795a6d70eb40a82566a0cce997474449cddfa45ffffe662a227c7092a3f 
hanwen@fedora:~/vc/go-fuse$ rm .jj/repo/op_heads/heads/58c3bc3179facd5d14211c62ee11182a4ea14ec2bca6ae8a3f990cfa90b1300a995b3def1e9a305e22ee8b38477ec9ec3e76368986466caf58c14f3c5d6b094c 
hanwen@fedora:~/vc/go-fuse$ jj log
Error: Unexpected error from backend: Error: Failed to read non-git metadata: failed to fill whole buffer
hanwen@fedora:~/vc/go-fuse$ ls -l  .jj/repo/op_heads/heads/58c3bc3179facd5d14211c62ee11182a4ea14ec2bca6ae8a3f990cfa90b1300a995b3def1e9a305e22ee8b38477ec9ec3e76368986466caf58c14f3c5d6b094c 
ls: cannot access '.jj/repo/op_heads/heads/58c3bc3179facd5d14211c62ee11182a4ea14ec2bca6ae8a3f990cfa90b1300a995b3def1e9a305e22ee8b38477ec9ec3e76368986466caf58c14f3c5d6b094c': No such file or directory
hanwen@fedora:~/vc/go-fuse$ ls -l  .jj/repo/op_heads/heads/96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33 
-rw-------. 1 hanwen hanwen 195 Sep  9 12:47 .jj/repo/op_heads/heads/96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33
hanwen@fedora:~/vc/go-fuse$ > .jj/repo/op_heads/heads/96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33 
hanwen@fedora:~/vc/go-fuse$ ls -l  .jj/repo/op_heads/heads/96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33 
-rw-------. 1 hanwen hanwen 0 Sep  9 12:48 .jj/repo/op_heads/heads/96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33
hanwen@fedora:~/vc/go-fuse$ jj log
Error: Unexpected error from backend: Error: Failed to read non-git metadata: failed to fill whole buffer
hanwen@fedora:~/vc/go-fuse$ rm  .jj/repo/op_heads/heads/96a525ef881f287e7757f43f085650fd7b4f83c68e212035796022aa86b2e86d45aac6d8879da11fb52363b5461e22c4c7466a5fea2709576f6d5ecb7cdd9e33 
hanwen@fedora:~/vc/go-fuse$ cp .jj/repo/op_store/operations/3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464   .jj/repo/op_heads/heads/
hanwen@fedora:~/vc/go-fuse$ jj log
Error: Unexpected error from backend: Error: Failed to read non-git metadata: failed to fill whole buffer
hanwen@fedora:~/vc/go-fuse$ jj-0.21 log
Failed to load commit index file 'cf48420db4e8047accd91c4f3e51ba30dc0122146ad11eb10c655ec346d8ca7625490a72cabb3c1e1c231d69ce16ea0113f8d83d817e85165a42a98e4c44b9b4' (maybe the format has changed): No such file or directory (os error 2). Reindexing...
Internal error: Failed to load the repo
Caused by:
1: Failed to index commits at operation 3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464
2: Failed to read non-git metadata
3: failed to fill whole buffer
hanwen@fedora:~/vc/go-fuse$ cp .jj/repo/op_store/operations/54d76eb5f6a20da79c32091d4a1ebb0e0e079af6f6e12fb060cd93d5024116a0a92753b8583f3c74d5c345f09c9be65d1d5e60e3191ac8b61b922f5833b90f5a   .jj/repo/op_heads/heads/
hanwen@fedora:~/vc/go-fuse$ rm .jj/repo/op_heads/heads/
3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464
54d76eb5f6a20da79c32091d4a1ebb0e0e079af6f6e12fb060cd93d5024116a0a92753b8583f3c74d5c345f09c9be65d1d5e60e3191ac8b61b922f5833b90f5a
hanwen@fedora:~/vc/go-fuse$ rm .jj/repo/op_heads/heads/54d76eb5f6a20da79c32091d4a1ebb0e0e079af6f6e12fb060cd93d5024116a0a92753b8583f3c74d5c345f09c9be65d1d5e60e3191ac8b61b922f5833b90f5a 
hanwen@fedora:~/vc/go-fuse$ jj log
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Other(ReadMetadata(IoError(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })))', lib/src/default_index_store.rs:245:42
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
hanwen@fedora:~/vc/go-fuse$ jj-0.21 log
Internal error: Failed to load the repo
Caused by:
1: Failed to index commits at operation 3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464
2: Failed to read non-git metadata
3: failed to fill whole buffer
hanwen@fedora:~/vc/go-fuse$ jj-0.
jj-0.15   jj-0.21   jj-0.7.0  
hanwen@fedora:~/vc/go-fuse$ ls -l  .jj/repo/op_heads/heads/
total 4
-rw-------. 1 hanwen hanwen 195 Sep  9 12:48 3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464
hanwen@fedora:~/vc/go-fuse$ >  .jj/repo/op_heads/heads/
bash: .jj/repo/op_heads/heads/: Is a directory
hanwen@fedora:~/vc/go-fuse$ >  .jj/repo/op_heads/heads/3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464 
hanwen@fedora:~/vc/go-fuse$ jj-0.21 log
Internal error: Failed to load the repo
Caused by:
1: Failed to index commits at operation 3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464
2: Failed to read non-git metadata
3: failed to fill whole buffer
hanwen@fedora:~/vc/go-fuse$ jj-0.21 --ignore-working-copy  op log -T id  
hanwen@fedora:~/vc/go-fuse$ jj-0.21 --ignore-working-copy  log
Internal error: Failed to load the repo
Caused by:
1: Failed to index commits at operation 3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464
2: Failed to read non-git metadata
3: failed to fill whole buffer
hanwen@fedora:~/vc/go-fuse$  rm .jj/repo/op_heads/heads/3efe2d30cd583982f41640a10f966baa93d08bd0028f4c4b383f54de8aa167f52e5dfecf6023c1748ca4fb755c05b97146d1a9a187f270a227a2eefd07ad6464
hanwen@fedora:~/vc/go-fuse$ >.jj/repo/op_heads/heads/54d76eb5f6a20da79c32091d4a1ebb0e0e079af6f6e12fb060cd93d5024116a0a92753b8583f3c74d5c345f09c9be65d1d5e60e3191ac8b61b922f5833b90f5a
hanwen@fedora:~/vc/go-fuse$ jj-0.21  log
Internal error: Failed to load the repo
Caused by:
1: Failed to index commits at operation 54d76eb5f6a20da79c32091d4a1ebb0e0e079af6f6e12fb060cd93d5024116a0a92753b8583f3c74d5c345f09c9be65d1d5e60e3191ac8b61b922f5833b90f5a
2: Failed to read non-git metadata
3: failed to fill whole buffer
hanwen@fedora:~/vc/go-fuse$ jj-0.15  log
thread 'main' panicked at lib/src/repo.rs:257:22:
called `Result::unwrap()` on an `Err` value: IndexReadError(IndexCommits { op_id: OperationId("54d76eb5f6a20da79c32091d4a1ebb0e0e079af6f6e12fb060cd93d5024116a0a92753b8583f3c74d5c345f09c9be65d1d5e60e3191ac8b61b922f5833b90f5a"), source: Other(ReadMetadata(TableStoreError(Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }))) })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
hanwen@fedora:~/vc/go-fuse$ jj-0.21  --ignore-working-copy log
Internal error: Failed to load the repo
Caused by:
1: Failed to index commits at operation 54d76eb5f6a20da79c32091d4a1ebb0e0e079af6f6e12fb060cd93d5024116a0a92753b8583f3c74d5c345f09c9be65d1d5e60e3191ac8b61b922f5833b90f5a
2: Failed to read non-git metadata
3: failed to fill whole buffer

hanwen avatar Sep 09 '24 10:09 hanwen

while trying to recover, it looks like some of the git data wasn't written with fsync(),

hanwen@fedora:~/vc/go-fuse$ find .git/objects/ -empty
.git/objects/14/77eae74176bafe96fcca4650e635f8c80fde31
.git/objects/35/a94a395cf102a80b78462d33a1e61d672cce0f
.git/objects/2b/11fef368bdd4a5a340a17a5ca85e2e334409d6
.git/objects/80/0ae0a71365ac4d6c19ac17b8f84794514f0d57
.git/objects/87/10cbdd77935071ef7b70c6eda96535e24e3359
hanwen@fedora:~/vc/go-fuse$ rm $(find .git/objects/ -empty)
rm: remove write-protected regular empty file '.git/objects/14/77eae74176bafe96fcca4650e635f8c80fde31'? y
rm: remove write-protected regular empty file '.git/objects/35/a94a395cf102a80b78462d33a1e61d672cce0f'? y
rm: remove write-protected regular empty file '.git/objects/2b/11fef368bdd4a5a340a17a5ca85e2e334409d6'? y
rm: remove write-protected regular empty file '.git/objects/80/0ae0a71365ac4d6c19ac17b8f84794514f0d57'? y
rm: remove write-protected regular empty file '.git/objects/87/10cbdd77935071ef7b70c6eda96535e24e3359'? y

hanwen avatar Sep 09 '24 11:09 hanwen

hanwen@fedora:~/vc/go-fuse$ touch .jj/repo/op_heads/heads/d291b16b2b2cdec761c961976542eedaeafc9cb537383312a5bec20742a99cae3148ca36ced4fe03eda64f9276989e88d17d62c68035da8bbe8ce5ef1cd68c67

That won't be enough. You also need to delete the other operation that was already present in that directory. Otherwise the next command you run will attempt to reconcile the two operations ("Concurrent modification detected, resolving automatically."), and since we know the other old was broken, it will fail.

Let us know if that still doesn't work.

martinvonz avatar Sep 09 '24 16:09 martinvonz

I did try and make sure some of the commands ran with just a single head, but it didn't help.

after seeing that the base git data was corrupted, I gave up and created a new repo. I didn't lose anything, as I had my WIP as a text diff and Gerrit changes.

Feel free to close (although the error message could probably be better.)

hanwen avatar Sep 09 '24 18:09 hanwen

FWIW I hit the same error message after a reboot with similar symptoms. I was able to reset head to 000000.. (the only other op from jj --ignore-working-copy op log -T id) but there wasn't much there and I ended up just recreating the jj repo (since it was at least colocated and the git repo wasn't corrupted).

Attaching the zipped .jj dir here for posterity, in case it helps anyone with troubleshooting: jj_vpm_bak.zip.

I do think it's important that jj gives some kind of more helpful error output for data loss cases even if we're not gonna be able to recover or prevent these kinds of repo corruption.

dbarnett avatar Oct 19 '24 18:10 dbarnett

I unfortunately also encountered this sort of issue after an unrelated kernel oops, but needed to gain access to the data in the repository.

After failing to roll back the op_heads/heads to a usable state as well, I've manually extracted my changes from the git store instead.

Here are some notes on what I've done, hopefully this can help someone else out. Although they might need some tweaking if the repo format changes.

Exact errors for easier search/comparison.

After the kernel oops:

> jj
Internal error: Failed to load an operation
Caused by:
1: Error when reading object d411a0653218a1a47662bc548637c940acd7922385484ac7e7021e3051717b955d66b62c886e55771dedc0bedb0e85c5a84463eb11284a2503c1d0a2e4d1098a of type operation
2: Invalid hash length (expected 64 bytes, got 0 bytes)

After rolling back op_heads to a known good op:

> jj
Internal error: Unexpected error from backend
Caused by:
1: Failed to read non-git metadata
2: Failed to load table segment '9dea43e70bbc83c029b2c073809694c2b2185ad7cc1089b72e96391ca231a91e8d6391b635759c5cee715d0a3364a2312a373fc092cc71a555325ab2c3afd249'
3: failed to fill whole buffer

Potential recovery steps

The following assumes you have a project directory called P containing the corrupted JJ repo.

  • Backup your project, make sure the backup contains the .jj directory, and ideally preserve file timestamps
  • Create a new directory (outside of the project), these steps use the name P.rec
  • Copy the git repo store P/.jj/repo/store/git to P.rec/.git
  • Make the git repo in P.rec non-bare, i.e. remove the bare key in .git/config

You now have access to the git repo that JJ stores parts of its data in. We are now interested in the commits that are marked for safe-keeping, they are kept in .git/refs/jj/keep. At the time of writing they file names match the commit hashes written in them.

You need to identify the commit(s) that contain changes that you are interested in. If you preserved your timestamps, using recent ones is probably a safe bet.

If you, like me, lost or ruined your timestamps, you could instead try to eyeball the history of each commit. Depending on the age of the JJ repo, and the amount of history in the git repo this may be more complicated.

For a young JJ repo with a short history it may be sufficient to use a script like this:

# fish
for c in (ls .git/refs/jj/keep/)
    echo "-- Log for $c --"
    git log --pretty='format:%h %ci %s' $c
    echo
end

After you've identified your commit, just check it out with git switch --detach <commit> and copy it to a different location for safe keeping.

If you hit an error like fatal: bad object refs/jj/keep/ed0fda877a0b5875eac25192698787d574feb02e, you've found a/the commit that was not properly written. The file is probably empty, and can be deleted to allow git to proceed.

SyrupThinker avatar Apr 03 '25 22:04 SyrupThinker

"Me too":

$ jj
Internal error: Failed to load an operation
Caused by:
1: Error when reading object 8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330 of type operation
2: Invalid hash length (expected 64 bytes, got 0 bytes)
$ find . -name 8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330 | xargs ls -l
-rw-------. 1 dhardy dhardy 0 Apr  7 09:10 ./.jj/repo/index/operations/8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330
-rw-r--r--. 1 dhardy dhardy 0 Apr  7 09:10 ./.jj/repo/op_heads/heads/8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330
-rw-------. 1 dhardy dhardy 0 Apr  7 09:10 ./.jj/repo/op_store/operations/8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330

Repo archive available on request (~24MiB).

It would be nice if all filesystem writes were atomic or recoverable...

I was eventually able to recover the repo.

First, I found several empty files under .git. Attempting to use them (next step) resulted in errors, so I deleted all those with hashes:

.git/branches
.git/refs/jj/keep/3c35afef543639cafe48af47e1923da32a647a0b
.git/refs/jj/keep/a3f52cb83379b0c628ad215a11a3a1d8d864813a
.git/refs/jj/keep/a221881d2603c747068637faf55e42b5b7a9ed10
.git/refs/jj/keep/fb0ad1e0c4ef00c62dd922e1a1b5489484ca9548
.git/objects/info
.git/objects/fb/0ad1e0c4ef00c62dd922e1a1b5489484ca9548
.git/objects/a3/f52cb83379b0c628ad215a11a3a1d8d864813a
.git/objects/a2/21881d2603c747068637faf55e42b5b7a9ed10
.git/objects/3c/35afef543639cafe48af47e1923da32a647a0b

git fsck also points out an erroneous head, so I deleted that too:

$ git fsck
Checking object directories: 100% (256/256), done.
Checking objects: 100% (46986/46986), done.
error: refs/heads/push-zsmyoumtzwww: invalid sha1 pointer a3f52cb83379b0c628ad215a11a3a1d8d864813a
error: HEAD: invalid reflog entry a3f52cb83379b0c628ad215a11a3a1d8d864813a
$ rm .git/refs/heads/push-zsmyoumtzwww

(There are also many dangling blobs/commits/trees; ignore those since they are normal with jj.)

Now, lets find the latest commit used by jj and restore that:

$ ls -ltr .git/refs/jj/keep/
# <snip>
-rw-r--r--. 1 dhardy dhardy 40 Apr  7 09:10 382e30b7e53fdd410128f1468f4e800021feebf7
$ git update-ref HEAD 382e30b7e53fdd410128f1468f4e800021feebf7
$ git status
HEAD detached from 4f6b1a34
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        modified:   crates/kas-core/Cargo.toml
        modified:   crates/kas-core/src/config/theme.rs

Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   crates/kas-core/Cargo.toml
        modified:   crates/kas-core/src/config/theme.rs
        modified:   crates/kas-resvg/Cargo.toml
$ git stash
Saved working directory and index state WIP on (no branch): 382e30b7 JJ: Description from the destination commit: Update async-global-executor, smol_str, ron

Great, git is now functional!

Now for jj. Lets check recent operations through the file system:

$ ls -ltr .jj/repo/op_store/operations
# <snip>
-rw-------. 1 dhardy dhardy 253 Apr  7 09:10 af7cdb32beed6539326779f9ffcdfd3e13685a7b142eeaf59a42afabf53434396d7679175d2a32fddb5763d83c62080cb0baeff7ca505813d1190f42d76a9cdc
-rw-------. 1 dhardy dhardy   0 Apr  7 09:10 474bb8b593f7bce25f9a4f9022cf31eb28c53337ab8c262ea1d499220fb077748f9544679334f4b7ed6f4a67458a78e42fa74bc2403ea58784e7f4647fb7e3f7
-rw-------. 1 dhardy dhardy   0 Apr  7 09:10 8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330

We can check these with jj op log --at-op $HASH. The last two entries have zero size and are not usable, so I deleted those and choose the third-last (af7cdb).

Next, lets try to check the status:

# <snip>
Failed to evaluate short-prefixes disambiguation revset
Unexpected error from store
Failed to read non-git metadata
Failed to load table segment 'a6db57f1a918103ac09393d5f84742d09c1ae77075e966d56df01ab6ee1a37e0268b09730f59ffe9bffadd81cb977b1324f3cbe0ba4bb4d20edd01ce747361a9'
failed to fill whole buffer

Hmm, lets investigate that:

$ find . -name a6db57f1a918103ac09393d5f84742d09c1ae77075e966d56df01ab6ee1a37e0268b09730f59ffe9bffadd81cb977b1324f3cbe0ba4bb4d20edd01ce747361a9 | xargs ls -l
-rw-------. 1 dhardy dhardy 0 Apr  7 09:10 ./.jj/repo/store/extra/a6db57f1a918103ac09393d5f84742d09c1ae77075e966d56df01ab6ee1a37e0268b09730f59ffe9bffadd81cb977b1324f3cbe0ba4bb4d20edd01ce747361a9
-rw-r--r--. 1 dhardy dhardy 0 Apr  7 09:10 ./.jj/repo/store/extra/heads/a6db57f1a918103ac09393d5f84742d09c1ae77075e966d56df01ab6ee1a37e0268b09730f59ffe9bffadd81cb977b1324f3cbe0ba4bb4d20edd01ce747361a9
$ find . -name a6db57f1a918103ac09393d5f84742d09c1ae77075e966d56df01ab6ee1a37e0268b09730f59ffe9bffadd81cb977b1324f3cbe0ba4bb4d20edd01ce747361a9 -delete
$ jj status --at-op af7cdb32
The working copy has no changes.
Working copy : qzsxrwpn hidden 77c8dff2 (empty) (no description set)
Parent commit: vnttzykm hidden f15f2ed1 Update async-global-executor, smol_str, ron

Okay, now to restore to that op. We can't use jj op restore $LAST_GOOD_OP since it attempt to create a transaction over the current repo state, which is corrupt. What we can do is to try and create a new commit on top:

$ jj new --at-op af7cdb32
Internal error: Failed to export refs to underlying Git repo
Caused by:
1: Git error
2: The reference "HEAD" should have content f15f2ed111b823a7b3aeaa16332bbd1fbf806632, actual content was 382e30b7e53fdd410128f1468f4e800021feebf7

Okay, we can fix that.

$ git switch --detach f15f2ed111b823a7b3aeaa16332bbd1fbf806632
$ jj new --at-op af7cdb32
$ jj status
Internal error: Failed to load an operation
Caused by:
1: Object 8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330 of type operation not found
2: No such file or directory (os error 2)
$ find .jj -name 8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330 | xargs ls -l
-rw-------. 1 dhardy dhardy 0 Apr  7 09:10 .jj/repo/index/operations/8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330
-rw-r--r--. 1 dhardy dhardy 0 Apr  7 09:10 .jj/repo/op_heads/heads/8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330
$ find .jj -name 8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330 -delete
$ jj status
Error: Could not read working copy's operation.
Hint: Run `jj workspace update-stale` to recover.
See https://jj-vcs.github.io/jj/latest/working-copy/#stale-working-copy for more information.
$ jj workspace update-stale
Failed to read working copy's current operation; attempting recovery. Error message from read attempt: Object 8f4800175776ae164edf3afa033e783707a780758152bcfebec26c05785dd6c3f3c192786ecbb7d1e5d909ef531e21aa93ef765ba3171e6125d8b3770f2b6330 of type operation not found
Created and checked out recovery commit 9ad5518051f6
Abandoned 5 commits that are no longer reachable.
Rebased 3 descendant commits off of commits rewritten from git
Working copy now at: pqopqosk 5e514439 (empty) RECOVERY COMMIT FROM `jj workspace update-stale`
Parent commit      : svmvrxxt 62522017 (empty) (no description set)
Added 0 files, modified 5 files, removed 0 files
Done importing changes from the underlying Git repo.
$ jj status
The working copy has no changes.
Working copy : pqopqosk 5e514439 (empty) RECOVERY COMMIT FROM `jj workspace update-stale`
Parent commit: svmvrxxt 62522017 (empty) (no description set)

Great, we have a working jj repo again! Well, sort-of.

$ jj log
@  pqopqosk [email protected] 2025-04-11 10:59:37 5e514439
│  (empty) RECOVERY COMMIT FROM `jj workspace update-stale`
○  svmvrxxt [email protected] 2025-04-11 10:59:37 git_head() 62522017
│  (empty) (no description set)
○  qzsxrwpn [email protected] 2025-04-11 10:59:37 4eb75739
│  (empty) (no description set)
│ ○  tlnpmvnk hidden [email protected] 2025-04-07 08:39:20 0c4d37e4
│ │  Replace fns Node::for_child, find_node
│ ○  pkprsonv hidden [email protected] 2025-04-07 08:39:20 4ccc057f
├─╯  Replace fn Widget::for_child_node with child_node
│ ×  nymxsuos hidden [email protected] 2025-04-07 08:35:25 1fa970ac conflict
├─╯  (no description set)
│ ○  rozprztz hidden [email protected] 2025-04-07 08:57:49 98cb7a90
│ │  Add lifetime parameter to kas::Window
│ ○  swkytvqu hidden [email protected] 2025-04-07 08:34:27 92ec1cf6
├─╯  Widget macros: rename gen parameter 'a to '_a
◆  votmkksk [email protected] 2025-04-06 14:41:48 master fa12711c
│  (empty) Merge pull request #487 from kas-gui/push-wpksnrpwtkyu
~
$ jj diff -r tln
Error: Revision `tln` doesn't exist
$ jj diff -r nymx
Error: Revision `nymx` doesn't exist
$ jj diff -r rozp
Error: Revision `rozp` doesn't exist

So something is up with those commits. Fortunately, we can still use the git hash:

$ jj new 1fa970ac

dhardy avatar Apr 07 '25 08:04 dhardy

I had a similar error after hard-shutdown (battery died) but with a tree:

$ jj                                                                                                                                             
Internal error: Failed to snapshot the working copy                                                      
Caused by: Invalid hash length for object of type tree (expected 20 bytes, got 0 bytes): 
$

Doing rm .jj/working_copy/tree_state helped for me.

alberand avatar May 27 '25 10:05 alberand

Same here. I was working within wsl2 and used a jj workspace out of the main jj repo. I am pretty sure I did a normal shutdown yesterday. Today, the working state was broken:

Internal error: Failed to load an operation
Caused by:
1: Error when reading object 9b22dda0f60fe04852d1d3120a9c311c496bf7e7c801ac94060927c979bd6f1ad4cfef0760a4fc6e879aaa4d723681aa5bf26d0d27b47285187d153298bd2386 of type operation
2: Invalid hash length (expected 64 bytes, got 0 bytes)

Sadly I wasn't able to restore the state of the workspace.

I understand that this might be a hard problem to solve. Maybe it would be easier to implement a command that attempts to restore a corrupted jj repo?

PointerDilemma avatar Jul 16 '25 09:07 PointerDilemma

Same here. Caused by unclean shutdown (blackout)

Internal error: Failed to load an operation
Caused by:
1: Error when reading object 09c0b74128f566103009a0baa236d2c462ea3c36d4b94e13872a44521192da428827b15600d076e2810257e93b7068f8815b559f4f2e804e28fd0f2d3d214d2b of type operation
2: Invalid hash length (expected 64 bytes, got 0 bytes)

0x4A4FRN avatar Aug 24 '25 15:08 0x4A4FRN

Yeah I see this very frequently on WSL2. I can always restore it, but it's pretty annoying and mechanical.

RossSmyth avatar Aug 25 '25 15:08 RossSmyth

Had the same problem after a kernel crash on Win11. Both .jj and .git were "corrupted". Git didn't recognize as a git repo. JJ kept complaining about invalid commit ids.

Followed the below procedure to get the .git and .jj repos functioning again:

  • Get git working

    git fsck --full --no-dangling
# delete 0-byte commit refs
# repeat till no errors

    git log now works

  • Get jj partially working

    remove 0-byte files at .jj/repo/op_heads/heads and .jj/repo/op_store/operations/ jj --ignore-working-copy op log --at-op <operation_id> now works

  • git branch on any dangling commits you don't want to lose

    find a commit with git log, git update-ref HEAD <commit_sha> git status to see details of commit git branch... if this is of interest repeat till dangling commits at head are branched

  • rm -rf .jj/

  • jj git init [--colocate]

  • jj workspace update-stale

Of course, the list above doesn't include the mandatory diagnostics:

  • spend ~6 hours figuring out what happened and eventually land on #4423 (after a few side quests into git-bash and Warp after struggling with pwsh ;-) )
  • spend ~2 hours trying to fix .jj/ (but it was fun going down the rabbit hole with hanwen martinvonz dhardy SyrupThinker et al!)
  • give up in despair and go with the nuclear option (Thanks dbarnett!)

I have run so many git/jj commands in the last 2 days that I forget exactly which ones helped. The list above is what I can extract from the shell history - so "buyer beware"!

rkaramc avatar Nov 08 '25 18:11 rkaramc