runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard

[Windows] Failure authenticating to Github via SSH

Open jacobsheahan opened this issue 6 months ago • 5 comments

Description

Our Windows builds are consistently failing ever since updating to image version: '20250617.1.0'. Previously were working using image version: '20250609.2.0' (and number of versions before this).

Git error. Command: `git clone --mirror [email protected]:<my repo> C:\Users\VssAdministrator\AppData\Local\Pub\Cache\_temp\dir7816cfc6`
stdout: 
stderr: Cloning into bare repository 'C:\Users\VssAdministrator\AppData\Local\Pub\Cache\_temp\dir7816cfc6'...
Read from remote host github.com: Connection reset by peer
client_loop: send disconnect: Connection reset by peer
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

exit code: 128

We do have a step that uses InstallSSHKey@0 task to install the SSH key. Notice that the version of the git tools was updated between the working version of runner and current version of runner.

Also note we are also building using macos-14 version '20250623.1377' and macos-13 version '20250616.1247' for iOS and Android that are working fine

Platforms affected

  • [x] Azure DevOps
  • [ ] GitHub Actions - Standard Runners
  • [ ] GitHub Actions - Larger Runners

Runner images affected

  • [ ] Ubuntu 22.04
  • [ ] Ubuntu 24.04
  • [ ] macOS 13
  • [ ] macOS 13 Arm64
  • [ ] macOS 14
  • [ ] macOS 14 Arm64
  • [ ] macOS 15
  • [ ] macOS 15 Arm64
  • [ ] Windows Server 2019
  • [x] Windows Server 2022
  • [x] Windows Server 2025

Image version and build link

20250617.1.0'

Is it regression?

Yes; 20250609.2.0

Expected behavior

Git able to clone to authenticated repo using SSH

Actual behavior

Git error. Command: `git clone --mirror [email protected]:<my repo> C:\Users\VssAdministrator\AppData\Local\Pub\Cache\_temp\dir7816cfc6`
stdout: 
stderr: Cloning into bare repository 'C:\Users\VssAdministrator\AppData\Local\Pub\Cache\_temp\dir7816cfc6'...
Read from remote host github.com: Connection reset by peer
client_loop: send disconnect: Connection reset by peer
fetch-pack: unexpected disconnect while reading sideband packet
fatal: early EOF
fatal: fetch-pack: invalid index-pack output

exit code: 128

Repro steps

See description

jacobsheahan avatar Jun 23 '25 19:06 jacobsheahan

Hi We are looking into this issue and will keep you updated here.

lakshminarayana02 avatar Jun 24 '25 04:06 lakshminarayana02

I got around it by downgrading git.

    steps:
      - name: Install Git 2.49.0 Portable
        run: |
          $url = "https://github.com/git-for-windows/git/releases/download/v2.49.0.windows.1/PortableGit-2.49.0-64-bit.7z.exe"
          Invoke-WebRequest -Uri $url -OutFile git-portable.exe
          Start-Process -Wait -FilePath ".\git-portable.exe" -ArgumentList "-y", "-oC:\tools\git\2.49.0"
          Remove-Item -Path .\git-portable.exe -Force
          echo "C:\tools\git\2.49.0\cmd" >> $env:GITHUB_PATH

      - name: Confirm Git usage
        run: git --version

housei avatar Jun 24 '25 04:06 housei

I got around it by downgrading git.

    steps:
      - name: Install Git 2.49.0 Portable
        run: |
          $url = "https://github.com/git-for-windows/git/releases/download/v2.49.0.windows.1/PortableGit-2.49.0-64-bit.7z.exe"
          Invoke-WebRequest -Uri $url -OutFile git-portable.exe
          Start-Process -Wait -FilePath ".\git-portable.exe" -ArgumentList "-y", "-oC:\tools\git\2.49.0"
          Remove-Item -Path .\git-portable.exe -Force
          echo "C:\tools\git\2.49.0\cmd" >> $env:GITHUB_PATH

      - name: Confirm Git usage
        run: git --version

Thanks for the suggestion - can confirm that this workaround did work for me!

jacobsheahan avatar Jun 24 '25 14:06 jacobsheahan

Hi @jacobsheahan - Please confirm if this issue has already been resolved on your end. Thank you!"

Prabhatkumar59 avatar Jun 26 '25 09:06 Prabhatkumar59

It's still happening today on our runners: windows-2022-8core_bf60cf62bc36 Version: 20250617.1.0

grzegorzkrukowski avatar Jun 27 '25 21:06 grzegorzkrukowski

Hi @jacobsheahan - Since we haven't heard back, we'll assume your issue is resolved as per your workaround and we will close this issue for now. Feel free to reach out to us for any other queries. Thanks.

Prabhatkumar59 avatar Jun 30 '25 06:06 Prabhatkumar59

@Prabhatkumar59 this also started happening consistently on windows-2025 as well the other day. So does not seem solved.

FWIW I was able to independently reproduce this in a VM with 2.50 and not 2.49. So I suggest the runner should downgrade until git-for-windows resolves the issue.

jamesdolan avatar Jun 30 '25 19:06 jamesdolan

I can confirm it's still happening. Downgrading manually on each runner doesn't seem like a right solution.

grzegorzkrukowski avatar Jul 01 '25 11:07 grzegorzkrukowski

@Prabhatkumar59 that is a joke to be honest, the guys at Microsoft/GitHub break the base image not for the first time. Then someone finds a small trick to workaround the bug in the base image and instead of fixing the base image the issue get closed.

Just fix the base image! That was not the first time that a random base image update (which I cannot prevent as far I know) breaks our builds. Because of such things we actually think about no to use the hosted runners because things break randomly.

rekire avatar Jul 07 '25 04:07 rekire

Is that fixed already on the latest version or we still need to keep the workaround active?

grzegorzkrukowski avatar Aug 21 '25 14:08 grzegorzkrukowski