dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827 Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution Pull request: https://github.com/dotnet/sdk/pull/39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827 Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1075586	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49459
1075041	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#49454
1074797	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49454
1074204	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49454
1073384	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49459
2732403	dotnet-sdk	Run FullFramework Tests	Log	#51011
2732285	dotnet-sdk	Run TestBuild Tests	Log	#50939
2732284	dotnet-sdk	Run TestBuild Tests	Log	#51000
2731748	dotnet-sdk	Run FullFramework Tests	Log	#50943
1070309	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49326
1069918	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49444
2731715	dotnet-sdk	Run FullFramework Tests	Log	#50938
1069692	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49421
1069571	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49284
1066660	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49284
1068620	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49408
1068588	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#49409
1068007	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49389
1067512	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49374
1067583	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49393
2729237	dotnet-sdk	Run FullFramework Tests	Log	#50855
1066921	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49375
1066696	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49374
1065747	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49290
1065794	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49357
1065782	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49288
1065766	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49306
1065706	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49352
1065689	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49329
1065749	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#49356
1065631	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49118
1064368	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49242
1060792	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49280
1065610	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49310
2728318	dotnet-sdk	🟣 Run ContainerBased Tests (debian12Amd64)	Log
1065477	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49332
1065487	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49306
1064445	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49290
1064363	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49239
1065543	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49288
1064753	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49306
1065101	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49288
1065102	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49330
1065083	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49336
1065159	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#48906
1065055	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49329
2728071	dotnet-sdk	🟣 Run TestBuild Tests	Log
1064899	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49348
1064897	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49346
1064858	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49308
1064808	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49315
1064765	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#48387
1064741	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49287
2727783	dotnet-sdk	🟣 Run TestBuild Tests	Log
1064729	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49311
1064585	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49306
1064373	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49335
1064242	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49287
1064372	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49336
1064474	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49306
1064303	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49166
1064298	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49334
2727354	dotnet-sdk	🟣 Run TestBuild Tests	Log
1064268	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49332
1064272	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49150
1064250	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49328
1064248	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49315
1064233	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49314
1064216	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#48906
1064222	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49312
1064208	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49330
1064206	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49329
1064207	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49186
1064212	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49284
1064194	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#48819
1064304	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49290
1063970	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49057
1064090	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49323
2727284	dotnet-sdk	🟣 Run ContainerBased Tests (debian12Amd64)	Log
1063209	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49306
1063972	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#49061
1063176	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49290
1063880	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49315
1064038	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49316
1063879	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49315
1063868	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49314
1063854	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49312
1063853	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49312
1063978	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#49134
1063957	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#45548
1063902	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#48906
1063897	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49150
1063768	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49311
1063737	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49310
1062477	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49013
1063788	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#49150
2727011	dotnet-sdk	🟣 Run ContainerBased Tests (debian12Amd64)	Log
1063683	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49309
1063686	dotnet/sdk	🟣 Run ContainerBased Tests	Log	dotnet/sdk#49310
1063536	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#48387
Displaying 100 of 179 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
2	15	179

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: https://github.com/dotnet/sdk/pull/41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR: https://github.com/dotnet/sdk/pull/39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.

[Forgind]

Ping @tmat This is still a serious pain point, currently preventing MSBuild from merging their code flow to 9.0.2xx. I already clicked rerun twice. I pointed to a PR that I think is likely to blame.

[tmat]

Fixes targeting 9.0.3xx: https://github.com/dotnet/sdk/pull/45999, https://github.com/dotnet/sdk/pull/46141

[nagilson]

@tmat Thank you for making some PRs to try to remedy this -- we are still seeing this in 9.0.2xx, 9.0.1xx, and main, 10 preview 2. Do your changes need to be ported?

[tmat]

They don't flow automatically to main?

Re 9.0.2xx, 9.0.1xx - these are product changes. Do they meet the servicing bar?

[nagilson]

They should flow to main, but either something went wrong and it didn't for those PRs, or the fix did not work, since it is still showing up in .net 10 preview 2: https://github.com/dotnet/sdk/pull/46983

I dont think they would meet the bar that, so good point. 9.0.3xx though, they probably do.

[tmat]

Might be something else causing failure in 10 P2. Do you have a link to the failed run? The PR is merged.

[baronfel]

This has spiked a lot in the last week - 83 hits out of ~140 for the month. @tmat would you be able to dig in and see if there's something fundamental that needs addressing?

[Forgind]

Did you ever look into https://github.com/dotnet/sdk/issues/40006#issuecomment-2311025564?

[tmat]

I can't access logs from the linked PRs:

[tmat]

@Forgind Yes, there is no issue with F#.

[tmat]

If anyone has a PR with dotnet-watch failure, please download the test logs and share them here. Helix links are not useful.