`git cinnabar fsck` reports bad files

[saschanaz]

> git cinnabar --version
0.5.9a
module-hash: 5e3702f55a47d0bbe5d819cb93d960888138b9be
helper-hash: 35b90b2bf83cfb025e256ca1d65dc8763437b517

> git cinnabar fsck
Checking 8 changeset heads
Loading 17316 manifests
Checking 23 manifest heads
Sha1 mismatch for file media/mozva/mozva.c
  revision 8e7f694721797cb82f6573f32c6d3ffb6ef15fac
  with parent 49aac12cc4dc6cc360de7c71fa93903341522857
Sha1 mismatch for file media/mozva/va/README
  revision 0250c7ce5720e7c10510d41ce30829da3a09fe1b
Sha1 mismatch for file media/mozva/va/va.h
  revision d2566169de6a0f824b56dedec361e05fe8f3c916
Sha1 mismatch for file media/mozva/va/va.patch
  revision 05a5c75ab6c1cab7c28571c2b2b2b8b5b6831463
Sha1 mismatch for file media/mozva/va/va_dec_av1.h
  revision b8bf4cc914f475fabfd505d223f7f8dca2863255
Sha1 mismatch for file media/mozva/va/va_dec_vp8.h
  revision bb3beb2542ff23bb0b1525d691fb3a87c5e76831
Sha1 mismatch for file media/mozva/va/va_dec_vp9.h
  revision 4feb8c50a635bb97369008c4a1f1ba44bfd7388c
Sha1 mismatch for file media/mozva/va/va_version.h
  revision 8017bcff5b3b54798c5408f2c983abb8e86f755b
Sha1 mismatch for file third_party/rust/clap/src/build/app/settings.rs
  revision 7d202d211c65ebc03593294d47c9c8589178592b
Sha1 mismatch for file third_party/rust/clap/src/build/arg/mod.rs
  revision 682fe035050186d92e7495025d49153174333946
Sha1 mismatch for file third_party/rust/clap/src/build/arg/settings.rs
  revision 51adf7ed600b68b0294f1df76af483227bf67675
Sha1 mismatch for file third_party/rust/clap/src/build/arg_group.rs
  revision 1eec6e8289428b9eb98d5a4dc9b2612325fd7437
Sha1 mismatch for file third_party/rust/clap/src/build/macros.rs
  revision a88060e7a1e2e4b949befa0178dcbe9426bdc9a5
Sha1 mismatch for file third_party/rust/terminal_size/README.md
  revision cf8539acb96fde89b45aa27555b29c3d0f1afd75
Checking 47265 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues

Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.

Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.

Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.

But those revision don't seem to actually exist?

[glandium]

Is it the same repo you fixed in #280? Can you create a git bundle the same way you did back then?

[glandium]

Can you do another one with git bundle create $file refs/cinnabar/metadata refs/cinnabar/checked refs/cinnabar/broken? (without --not ...)

[saschanaz]

Sent it 👍

[glandium]

This time, your repo is more busted than last time, but it's still the same root cause (#207). I'm bummed that the situation was not detected.

You seem to have a setup that amplified the problem, creating a diverging history from gecko-dev. I suspect you have a pref like cinnabar.data or remote.something.cinnabar-data set to always. I wouldn't recommend you use that.

You can fix your repo with git cinnabar rollback b1fe9b8493dbb38038c230f2fb346bfa803a60fe and re-pulling, but any local branch you have will need to be rebased.

I think you're the first person to whom this happened twice, so if you don't mind, I'd like to ask you to set something up after I work on some debugging code.

[saschanaz]

You seem to have a setup that amplified the problem, creating a diverging history from gecko-dev. I suspect you have a pref like cinnabar.data or remote.something.cinnabar-data set to always. I wouldn't recommend you use that.

No, this is all I have:

> git config --get-regexp cinnabar
cinnabar.version-check 1643634802
cinnabar.helper C:\Users\sasch\Documents\GitHub\git-cinnabar\git-cinnabar-helper.exe
cinnabar.fsck 1637682708

I think you're the first person to whom this happened twice, so if you don't mind, I'd like to ask you to set something up after I work on some debugging code.

👍 Feel free to ping me whenever it's ready.

[saschanaz]

During rebase...

> git pull
WARNING Mercurial libraries not found. Falling back to experimental native access.
Reading 1836 changesets
Reading and importing 1830 manifests
Reading and importing 3114 revisions of 1130 files
Importing 1836 changesets
Unpacking objects: 100% (40/40), 58.72 KiB | 7.34 MiB/s, done.
ERROR Error in file b'f917b11e765e626d9459c18cc3771e4decd75880'
ERROR Error in file b'8fc2e6a72551b0e749377adf2af35cdbd14c94ff'
Checking 1417 imported file root and head revisions
ERROR Object of type bytes is not JSON serializable
Run the command again with `git -c cinnabar.check=traceback <command>` to see the full traceback.
fatal: Unsupported command: error
fast-import: dumping crash report to .git/fast_import_crash_21684
fatal: error while running fast-import

> git cinnabar fsck
Checking 4 changeset heads
Loading 17606 manifests
Checking 21 manifest heads
Sha1 mismatch for file testing/web-platform/meta/cookies/attributes/secure.https.html.ini
  revision 8fc2e6a72551b0e749377adf2af35cdbd14c94ff
Sha1 mismatch for file gfx/angle/checkout/third_party/vulkan-deps/vulkan-headers/src/include/vulkan/vulkan_core.h
  revision f917b11e765e626d9459c18cc3771e4decd75880
Checking 48228 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues

Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.

Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.

Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.

👀 Should I send one again?

The crash report has nothing interesting:

fast-import crash report:
    fast-import process: 21684
    parent process     : 1
    at 2022-02-01 12:06:15 +0000

fatal: Unsupported command: error

Most Recent Commands Before Crash
---------------------------------
* error

Active Branch LRU
-----------------
    active_branches = 0 cur, 5 max

  pos  clock name
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Inactive Branches
-----------------

Marks
-----

-------------------
END OF CRASH REPORT

[glandium]

ERROR Error in file b'f917b11e765e626d9459c18cc3771e4decd75880'
ERROR Error in file b'8fc2e6a72551b0e749377adf2af35cdbd14c94ff'
Checking 1417 imported file root and head revisions
ERROR Object of type bytes is not JSON serializable

Is it the first time you see something like this?

> git config --get-regexp cinnabar
cinnabar.version-check 1643634802
cinnabar.helper C:\Users\sasch\Documents\GitHub\git-cinnabar\git-cinnabar-helper.exe
cinnabar.fsck 1637682708

How do you push to try?

[saschanaz]

Is it the first time you see something like this?

I believe I saw it at least once before this 🤔

How do you push to try?

I use ./mach try.

[glandium]

I believe I saw it at least once before this thinking

I wish you had reported it back then because it showed that the diagnostic code for #207 worked but failed to report the problem correctly with python3 :-/

I use ./mach try

Do you maybe git cinnabar fetch from try?

+1 Feel free to ping me whenever it's ready.

Can you update git-cinnabar to latest master, run git config cinnabar.check unbundler, then start again from https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1026529368 ? If it doesn't break again on that update, like it did in https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1026774885, can you keep that cinnabar.check value set until the next time it happens, at which time it will hopefully give you some instructions.

[saschanaz]

I wish you had reported it back then because it showed that the diagnostic code for #207 worked but failed to report the problem correctly with python3 :-/

Oops, sorry 😬

Do you maybe git cinnabar fetch from try?

TIL it exists, but I think I never fetch from try.

Can you update git-cinnabar to latest master, run git config cinnabar.check unbundler, then start again from #295 (comment) ?

Will do that 👍 (Edit: It didn't break)

[glandium]

TIL it exists, but I think I never fetch from try.

Something doesn't line up. When you push to try, can you check whether git rev-parse refs/cinnabar/metadata returns the same thing before and after you do the push?

[saschanaz]

> git pull --rebase origin bookmarks/central:main
WARNING Mercurial libraries not found. Falling back to experimental native access.
Reading 299 changesets
Reading and importing 299 manifests
Reading and importing 16 revisions of 12 filesfatal: Malformed file chunk for a59a87e470eca653bd2a0c76662372c9e5e1fe7b
fast-import: dumping crash report to .git/fast_import_crash_9400
Reading and importing 264 revisions of 249 files
Traceback (most recent call last):
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1003, in run
fatal: Unsupported command: error
f    retcode = func(args)
ast-import: dumping crash report to .git/fast_import_crash_17980
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\git-remote-hg", line 53, in main
    helper.run()
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\remote_helper.py", line 87, in run
    self.import_(*args)
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\remote_helper.py", line 422, in import_
    getbundle(self._repo, self._store, heads,
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 1071, in getbundle
    apply_bundle(store)
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 893, in __call__
    for rev_chunk in progress_enum(
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 224, in progress_enum
    for count, item in enum_iter:
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 869, in enumerate_files
    for count_chunks, (name, chunk) in enumerate(iterator, start=1):
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\hg\repo.py", line 837, in iter_files
    fh.write(chunk)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\git-remote-hg", line 59, in <module>
    run(main, sys.argv[1:])
fatal: error while running fast-import
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1012, in run
    message = message or ', '.join(
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 1013, in <genexpr>
    fsdecode(a) for a in getattr(e, 'args', []))
  File "C:\Users\sasch\AppData\Local\Programs\Python\Python310\lib\os.py", line 822, in fsdecode
    filename = fspath(filename)  # Does type-checking of `filename`.
TypeError: expected str, bytes or os.PathLike object, not int
Exception ignored in atexit callback: <bound method GitHgHelper.close of <class 'cinnabar.helper.GitHgHelper'>>
Traceback (most recent call last):
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\helper.py", line 513, in close
    super(GitHgHelper, self).close()
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\helper.py", line 108, in close
    self._helper.wait()
  File "C:\Users\sasch\Documents\GitHub\git-cinnabar\cinnabar\util.py", line 835, in wait
    fh.close()
OSError: [Errno 22] Invalid argument

> git cinnabar fsck
Checking 3 changeset heads
Loading 331 manifests
Checking 3 manifest heads
Sha1 mismatch for file toolkit/components/glean/tests/browser/browser_fog_gmp.js
  revision 61b8f076564ff8c1f2b16b5438b021fe152827e8
  with parent 56b7e3dc829cecaac0f860c2479d90babd9a5f71
Sha1 mismatch for file toolkit/components/glean/tests/browser/browser_fog_socket.js
  revision d956b2c06e402732a8548180afde2e1b9176fa53
  with parent de7589f6616d03e269aeeb5e0509354e136f41a1
Sha1 mismatch for file devtools/client/netmonitor/docs/architecture.md
  revision 2e0a5c33feaa57639a305f1c10eaa2789d17592f
  with parent 5d7e2850f40bc245751a4388f7435269f675c888
Checking 1232 files
Your git-cinnabar repository appears to be corrupted.
Please open an issue, with the information above, on
https://github.com/glandium/git-cinnabar/issues

Then please try to run `git cinnabar rollback --fsck` to restore last known state, and to update from the mercurial repository.

Please note this may affect the commit sha1s of mercurial changesets, and may require to rebase your local branches.

Alternatively, you may start afresh with a new clone. In any case, please keep this corrupted repository around for further debugging.

👀

[saschanaz]

Hmm, I ran git cinnabar fsck --force and got tons of mismatches 🤔, I wonder how it happened...

[glandium]

did you git cinnabar rollback b1fe9b8493dbb38038c230f2fb346bfa803a60fe as per https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1026529368 ?

[saschanaz]

I did git cinnabar rollback --fsck instead and thought it should be equivalent 👀 (as it did require rebasing.)

[glandium]

Kind of. I'm baffled you didn't see the other error message before that...

[saschanaz]

Which error message?

[glandium]

https://github.com/glandium/git-cinnabar/blob/6ee02238b841b67f63b5d4d5dbc5cf13f0de7dce/cinnabar/githg.py#L1324-L1336

[saschanaz]

No, haven't seen that one so far, sorry 🥲

[glandium]

Do you still have the repo from https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1031561207 ?

[saschanaz]

I don't but I made a gitbundle from it, so maybe I can say I do?

[glandium]

Is it a different clone from the previous one?

[saschanaz]

Same clone but after git cinnabar rollback (and right after the errors)

[glandium]

That doesn't sound true. The bundle you sent me has a bustage that started in august (which explains why you didn't get the bustage detection error), and doesn't contain b1fe9b8493dbb38038c230f2fb346bfa803a60fe

[saschanaz]

I mean right after https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1031561207, I made the bundle after this as I expected your request to share it, 🤔

Weird that it still had the bustage, I guess git cinnabar rollback --fsck didn't really work then? (I wonder that's why I saw https://github.com/glandium/git-cinnabar/issues/295#issuecomment-1031598598)

[glandium]

But it's a different clone than the one you opened this issue for, right?

[saschanaz]

It was still the same clone. I didn't do git cinnabar reclone, just the rollback.

[glandium]

So, this exposes a flaw in git cinnabar fsck, which is that it doesn't look at all the files that ever existed, but only those referenced by the current heads... So some files could still be broken, and, in your case, there were broken files in metadata older than b1fe9b8493dbb38038c230f2fb346bfa803a60fe. You'd probably be better off with a reclone at this point.

[glandium]

You do still seem to have hit #207 several times, so please keep cinnabar.check set to unbundler.

[saschanaz]

Yeah, I had to reclone to fix the broken pipe error. I'll report again if it happens again.