"mach try fuzzy" caused crash with signal 6 (`Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" })

[whimboo]

I'm running with artifact builds and wanted to push a build to try that has to generate full builds. As such --no-artifact has been used. But the push then stalled and after around 10 minutes failed with a crash:

mach try fuzzy --no-artifact                                                       git:(geckodriver_fission↑1|…7
estimates: Runs 62 tasks (22 selected, 40 dependencies)
estimates: Total task duration 12:30:53
estimates: In the top 52% of durations
estimates: Should take about 1:22:23 (Finished around 2023-01-31 14:50)
Bundling 2 changesets
Bundling 2 manifests
Bundling 4 revisions of 4 files
remote: waiting for lock on working directory of /repo/hg/mozilla/try held by process '28690' on host 'hgssh1.dmz.mdc1.mozilla.com/effffffc'
remote: abort: working directory of /repo/hg/mozilla/try: timed out waiting for lock held by 'hgssh1.dmz.mdc1.mozilla.com/effffffc:7178'
fatal: called `Result::unwrap()` on an `Err` value: Error { kind: UnexpectedEof, message: "failed to fill whole buffer" }
Run the command again with `git -c cinnabar.check=traceback <command>` to see the full traceback.
error: git-remote-hg died of signal 6
error: failed to push some refs to 'hg::ssh://hg.mozilla.org/try'
Error running mach:

    ['try', 'fuzzy', '--no-artifact']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file try| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '('/opt/homebrew/bin/git', 'push', 'hg::ssh://hg.mozilla.org/try', '+HEAD:refs/heads/branches/default/tip')' returned non-zero exit status 1.

  File "/Users/henrik/code/gecko/tools/tryselect/mach_commands.py", line 338, in try_fuzzy
    return run(command_context, **kwargs)
  File "/Users/henrik/code/gecko/tools/tryselect/mach_commands.py", line 196, in run
    return mod.run(**kwargs)
  File "/Users/henrik/code/gecko/tools/tryselect/selectors/fuzzy.py", line 248, in run
    return push_to_try(
  File "/Users/henrik/code/gecko/tools/tryselect/push.py", line 228, in push_to_try
    vcs.push_to_try(commit_message, allow_log_capture=allow_log_capture)
  File "/Users/henrik/code/gecko/python/mozversioncontrol/mozversioncontrol/__init__.py", line 739, in push_to_try
    subprocess.check_call(cmd, cwd=self.path)
  File "/Users/henrik/.pyenv/versions/3.9.10/lib/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)

[glandium]

I guess I should put a note here, better late than never. The relevant error is "abort: working directory of /repo/hg/mozilla/try: timed out waiting for lock held by 'hgssh1.dmz.mdc1.mozilla.com/effffffc:7178'", and git-cinnabar doesn't handle errors very well. So practically speaking, it's not a problem with git-cinnabar, but its lack of nice error handling is why I'm keeping this open.