sdk icon indicating copy to clipboard operation
sdk copied to clipboard

GenerateBundle Task fails writing to singlefilehost.exe

Open dsplaisted opened this issue 3 months ago • 14 comments

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1146095 Build error leg or test failing: Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution Pull request: https://github.com/dotnet/sdk/pull/50648

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "The process cannot access the file '.*singlefilehost\\.exe' because it is being used by another process",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1146095 Error message validated: [The process cannot access the file '.*singlefilehost\.exe' because it is being used by another process] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 9/12/2025 3:27:55 PM UTC

Report

Build Definition Test Pull Request
1218206 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51847
1216834 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51847
1213145 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51847
1212157 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51847
1212001 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51847
1210799 dotnet/sdk Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishIncrementally.Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishIncrementally.It_cleans_between_renames dotnet/sdk#51847
1210165 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51847
1209987 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51727
1209843 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51857
1209882 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51068
1209614 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51854
1209251 dotnet/sdk Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution dotnet/sdk#51817
1209276 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51847
1209103 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51836
1208914 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51614
1208863 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51828
1208905 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51560
1208327 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51817
1208519 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51614
1206490 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51790
1208315 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51812
1208215 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51817
1208193 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51812
1208181 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51801
1201304 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51697
1208052 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51727
1208054 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51812
1206495 dotnet/sdk Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishASingleFileApp.Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishASingleFileApp.It_can_disable_cetcompat dotnet/sdk#51792
1207838 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51727
1207787 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51801
1207162 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51801
1207260 dotnet/sdk Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution dotnet/sdk#51798
1206475 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51791
1206509 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51723
1206429 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51762
1206398 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51411
1206404 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51790
1206341 dotnet/sdk Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution dotnet/sdk#51411
1206192 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51788
1205093 dotnet/sdk Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution dotnet/sdk#51724
1205641 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51781
1205233 dotnet/sdk Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution dotnet/sdk#51760
1204153 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51760
1204961 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51411
1204812 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51747
1204493 dotnet/sdk Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution dotnet/sdk#51747
1204143 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51758
1204019 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51747
1204000 dotnet/sdk Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishIncrementally.Microsoft.NET.Publish.Tests.GivenThatWeWantToPublishIncrementally.It_cleans_between_renames dotnet/sdk#51747
1203943 dotnet/sdk Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution dotnet/sdk#51727
1203612 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51727
1203143 dotnet/sdk Microsoft.NET.Publish.Tests.dll.5.WorkItemExecution dotnet/sdk#51724
1203335 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51723
1203182 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51108
1203220 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51723
1202929 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51733
1202989 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51724
1202916 dotnet/sdk Microsoft.NET.Publish.Tests.dll.6.WorkItemExecution dotnet/sdk#51732

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 58

dsplaisted avatar Sep 12 '25 15:09 dsplaisted

@dotnet/illink @agocke We are seeing failures where the GenerateBundle task is failing because it cannot access the singlnefilehost.exe file. We added retry logic because we thought maybe the issues were caused by defender. I thought that helped, but now the issue is happening pretty consistently, so it may be a product issue.

Interestingly, it seems to only fail on Mac OS and Linux when bundling a Windows app. I can't remember if it was failing on Windows before we added the retry logic or not.

Can someone investigate this?

I'm also wondering whether we should merge this codeflow PR which is blocked by the failure. That's where I've seen the failure most consistently recently, so it's possible there's a code change coming in that made the problem worse.

dsplaisted avatar Sep 12 '25 16:09 dsplaisted

@jtschuster can you take a look?

agocke avatar Sep 13 '25 02:09 agocke

Ping @jtschuster, any progress on this? I just noticed it in https://github.com/dotnet/sdk/pull/50876

dsplaisted avatar Sep 18 '25 18:09 dsplaisted

I'm going to load balance a bit. @sbomer can you look instead?

agocke avatar Sep 18 '25 19:09 agocke

Still investigating, but I am able to repro locally on a mac, but only on the first run of the test. Subsequent runs use the same path for the tmp project and don't repro. It's also interesting that it's failing on a read and not a write.

jtschuster avatar Sep 18 '25 19:09 jtschuster

@jtschuster Any update on this? This is impacting a bunch of our CI runs.

dsplaisted avatar Sep 22 '25 17:09 dsplaisted

Still investigating exactly what's going on, but my hunch is that there's an async memory map flush that keeps the apphost mapped from CreateAppHost. In particular it looks like this todo might fix the issue: https://github.com/dotnet/runtime/blob/b8c39cc4d102518f8608df70e6e3cb7a5c10d48d/src/installer/managed/Microsoft.NET.HostModel/AppHost/HostWriter.cs#L172 The ResourceUpdater.Update() maps the destination apphost back into memory, and if there's async flushing after Dispose(), we could run into issues.

As a workaround, it looks like adding --disable-build-servers to the PublishCommand.Execute() method does something to make the issue fail to repro for both me and @sbomer. I'm not sure why, though.

jtschuster avatar Sep 22 '25 22:09 jtschuster

Thank you @jtschuster !

baronfel avatar Sep 25 '25 23:09 baronfel

Going to leave this open for Build Analysis until the runtime code flows

jtschuster avatar Sep 26 '25 04:09 jtschuster

Hmm maybe I shouldn't have used that wording in the commit

jtschuster avatar Sep 26 '25 15:09 jtschuster

This is also impacting the main branch which is 11 - https://github.com/dotnet/sdk/pull/51074 do we need to port it there?

nagilson avatar Oct 01 '25 21:10 nagilson

It looks like the commit is held up on flow to dotnet/dotnet https://github.com/dotnet/dotnet/pull/2627

It did make it to 10.0.1xx though: https://github.com/dotnet/dotnet/blob/release/10.0.1xx/src/runtime/src/installer/managed/Microsoft.NET.HostModel/ResourceUpdater.cs#L334-L345

jtschuster avatar Oct 01 '25 21:10 jtschuster

I am experiencing this on .NET 10 RC 1, specifically on Linux publishing a Windows application.

EDIT: RC 2 still has the same issue, even with the retry logic

RDI-Blake avatar Oct 14 '25 11:10 RDI-Blake

@RDI-Blake Thanks for the heads up. It looks like the fix just missed RC2, but it should be ready for the full release. If you're able to use one of the 10.0.100 daily builds do you still hit the failure?

jtschuster avatar Oct 16 '25 21:10 jtschuster