Figure out why pre-commit corrupts git repo

[adeebshihadeh]

pre-commit somewhat consistently corrupts my git repo on pre-commit init. It can be either pre-commit or one of the hooks we use. Goal is to identify why this happens and ideally fix it if possible.

Here's the latest occurrence for me, on the panda repo in this case (so it's a hook that openpilot has in common with panda).

batman:panda$ git commit -m "add assert function"
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/astral-sh/ruff-pre-commit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/usr/lib/git-core/git', 'diff', '--staged', '--name-only', '--no-ext-diff', '-z', '--diff-filter=ACMRTUXB')
return code: 128
stdout: (none)
stderr:
    Warning: Permanently added 'github.com' (ECDSA) to the list of known hosts.
    fatal: remote error: upload-pack: not our ref 8e00810bd7c4a26c8009324ed5215f99c22f1a1f
    Warning: Permanently added 'github.com' (ECDSA) to the list of known hosts.
    fatal: remote error: upload-pack: not our ref 8e00810bd7c4a26c8009324ed5215f99c22f1a1f
    fatal: unable to read 8e00810bd7c4a26c8009324ed5215f99c22f1a1f
Check the log at /home/batman/.cache/pre-commit/pre-commit.log
batman:panda$ git status
Warning: Permanently added 'github.com' (ECDSA) to the list of known hosts.
fatal: remote error: upload-pack: not our ref 8e00810bd7c4a26c8009324ed5215f99c22f1a1f
Warning: Permanently added 'github.com' (ECDSA) to the list of known hosts.
fatal: remote error: upload-pack: not our ref 8e00810bd7c4a26c8009324ed5215f99c22f1a1f
fatal: unable to read 8e00810bd7c4a26c8009324ed5215f99c22f1a1f

[adeebshihadeh]

Happened again on a completely different computer :(

adeeb:panda$ git commit -m "no serial sometimes?"
[INFO] Initializing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Initializing environment for https://github.com/pre-commit/mirrors-mypy:git+https://github.com/numpy/numpy-stubs,types-requests,types-atomicwrites,types-pycurl.
[INFO] Initializing environment for https://github.com/astral-sh/ruff-pre-commit.
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/astral-sh/ruff-pre-commit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
An unexpected error has occurred: CalledProcessError: command: ('/usr/lib/git-core/git', 'diff', '--staged', '--name-only', '--no-ext-diff', '-z', '--diff-filter=ACMRTUXB')
return code: 128
stdout: (none)
stderr:
    fatal: unable to read 8e00810bd7c4a26c8009324ed5215f99c22f1a1f
Check the log at /home/adeeb/.cache/pre-commit/pre-commit.log

EDIT:

Another data point: tried on that same PC as above for the msgq submodule and it didn't repro:

adeeb:msgq_repo$ touch abc
adeeb:msgq_repo$ git add .
adeeb:msgq_repo$ git commit -m "will it corrupt???"
[INFO] Initializing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Initializing environment for https://github.com/astral-sh/ruff-pre-commit.
[INFO] Installing environment for https://github.com/pre-commit/mirrors-mypy.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/astral-sh/ruff-pre-commit.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
[INFO] Installing environment for https://github.com/codespell-project/codespell.
[INFO] Once installed this environment will be reused.
[INFO] This may take a few minutes...
check python ast.....................................(no files to check)Skipped
check yaml...........................................(no files to check)Skipped
check that executables have shebangs.................(no files to check)Skipped
check that scripts with shebangs are executable......(no files to check)Skipped
mypy.................................................(no files to check)Skipped
ruff.................................................(no files to check)Skipped
cppcheck.............................................(no files to check)Skipped
cpplint..............................................(no files to check)Skipped
codespell............................................(no files to check)Skipped
[master e17598b] will it corrupt???
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 abc

[StefanLekanic-SYRMIA]

@adeebshihadeh Git status is aware of this problem. Additionally, if you run git fsck --full, you will get a list of missing blobs and dangling commits. You can clear the cache with the following commands:

git rm -r -f --cached . git reset --hard

After this, I was able to commit again.

[pantew869]

It maybe caused by mypy.

[pantew869]

@adeebshihadeh, Do https://github.com/commaai/rednose/pull/44 and other PRs count as a solution to this?

[adeebshihadeh]

Does it fix this? Do you have a reliable repro and can show it doesn't happen anymore after those PRs?

[pantew869]

Does it fix this? Do you have a reliable repro and can show it doesn't happen anymore after those PRs?

Yes, pre-commit broke submodules that used numpy-stubs because it cloned the numpy-stubs repo into the submodule.

[adeebshihadeh]

Cool, if you post a before/after that shows the repro, the bounty is yours!

[pantew869]

Cool, if you post a before/after that shows the repro, the bounty is yours!

Are screenshots enough?

[pantew869]

Before: After:

p.s it's a vm @adeebshihadeh

[adeebshihadeh]

Nice, bounty is yours. Open a ticket at comma.ai/support to claim.