runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard
verified publisher icon actions

windows-2025: Intermittent AccessViolationException in some long-running PowerShell steps

Open sharpjs opened this issue 3 months ago • 2 comments

Description

My team's main AzDO build-test-deploy pipeline that fails intermittently, about once every ten runs, when using the windows-2025 image. The failure does not occur with windows-2022. The error message is always some variation on the following:

Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

The error occurs after the pipeline has been running for some time (tens of minutes). It can occur in different kinds of steps. It most frequently occurs in any of several pwsh steps that run dotnet test. However it also occurs in an AzurePowerShell@5 step that runs sqlpackage.exe. We have only seen the failure in longer-running steps that use PowerShell and that launch executables from PowerShell. We have not seen the failure in longer-running PowerShell steps that are pure PowerShell.

The failure occurs equally on Microsoft hosted agents and on an Azure Managed DevOps Pool.

We have been watching and waiting for months in hope that the problem would just go away, but it has not. After a teammate got frustrated on his fourth consecutive error, we made the decision today to switch that pipeline back to windows-2022 and report a bug.

Running dotnet test. It happens both in test suites that target .NET Framework and those that target modern .NET. Image

When running sqlpackage.exe: Image

Platforms affected

  • [x] Azure DevOps
  • [ ] GitHub Actions - Standard Runners
  • [ ] GitHub Actions - Larger Runners

Runner images affected

  • [ ] Ubuntu 22.04
  • [ ] Ubuntu 24.04
  • [ ] macOS 13
  • [ ] macOS 13 Arm64
  • [ ] macOS 14
  • [ ] macOS 14 Arm64
  • [ ] macOS 15
  • [ ] macOS 15 Arm64
  • [ ] macOS 26 Arm64
  • [ ] Windows Server 2019
  • [ ] Windows Server 2022
  • [x] Windows Server 2025

Image version and build link

20250929.44.1

Is it regression?

windows-2022

Expected behavior

The described error does not occur.

Actual behavior

The described error occurs, causing the pipeline run to fail.

Repro steps

  1. Magically obtain access to our pipeline.
  2. Run the pipeline repeatedly until the error described above occurs.

sharpjs avatar Oct 09 '25 19:10 sharpjs

Hey @sharpjs, could you please provide repro steps or a sample project for us to troubleshoot the issue? Additionally, have windows-2025 agents always been this way or it happened after certain image version update?

shamil-mubarakshin avatar Oct 10 '25 08:10 shamil-mubarakshin

I can try to put together a fake pipeline similar to the one causing the issue, in an attempt to repro. If you are an employee or contractor with Microsoft, it might be doable to just give you access to the actual pipeline.

The problem has occurred with windows-2025 for months, ever since we started trying to use it.

sharpjs avatar Oct 10 '25 17:10 sharpjs