dotnet-watch tests failing

[dsplaisted]

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=629827 Build error leg or test failing: dotnet-watch.Tests.dll.1.WorkItemExecution Pull request: https://github.com/dotnet/sdk/pull/39888

Error Message

##[error].packages/microsoft.dotnet.helix.sdk/8.0.0-beta.24177.1/tools/Microsoft.DotNet.Helix.Sdk.MultiQueue.targets(89,5): error : (NETCORE_ENGINEERING_TELEMETRY=Test) Work item dotnet-watch.Tests.dll.1 in job 82728a56-9430-4356-ab82-1c21670792a7 has failed

{
  "ErrorMessage": "",
  "ErrorPattern": "Work item dotnet-watch\\.Tests\\.dll\\.1 in job [a-z0-9\\-]+ has failed",
  "BuildRetry": true,
  "ExcludeConsoleLog": true
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=629827 Error message validated: [Work item dotnet-watch\.Tests\.dll\.1 in job [a-z0-9\-]+ has failed] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/4/2024 9:51:10 AM UTC

Report

Build	Definition	Step Name	Console log	Pull Request
1230968	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52135
1230151	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52135
1229849	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#52148
2859464	dotnet-sdk	Run FullFramework Tests	Log	#56049
2859463	dotnet-sdk	Run TestBuild Tests	Log	#56048
1229178	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52053
1227774	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52053
1227484	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51307
1225970	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#52099
1225956	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#52098
1225955	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#52097
1227598	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52053
1226921	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51088
1226175	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52028
2857861	dotnet-sdk	Run TestBuild Tests	Log	#55976
2857862	dotnet-sdk	Run FullFramework Tests	Log	#55975
1224967	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52028
1224826	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#52007
2856664	dotnet-sdk	Run TestBuild Tests	Log	#55853
1223955	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51307
1223285	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51307
1222286	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51727
1222071	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51068
1222239	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51068
1221516	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#52007
1221157	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52033
1221118	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#52013
2854049	dotnet-sdk	Run TestBuild Tests	Log	#55844
1220569	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51068
1220056	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51881
1219988	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#52008
1219768	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51881
1219830	dotnet/sdk	Run FullFramework Tests	Log	dotnet/sdk#52006
1219822	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#52005
1219420	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51632
1218206	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1219523	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51419
1218232	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51935
1218297	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51985
1216834	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1216781	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51910
1216911	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51935
1216549	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51624
1216353	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51891
1215855	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51935
1213145	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1215739	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51068
1215251	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51947
1214650	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51942
1214670	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51935
1213624	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51881
1213293	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51881
1213207	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51918
1212185	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51881
1212157	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1212146	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51857
1212001	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1210799	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1210165	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1210104	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51861
1209844	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51826
1209743	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51826
1209507	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1209276	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51847
1209240	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51844
1209262	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51846
1209239	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51843
1208208	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51816
1208052	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51727
1207838	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51727
1206878	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#50864
1206801	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51419
1206632	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51794
1206490	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51790
1205092	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51698
1205101	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51705
1203943	dotnet/sdk	🟣 Run TestBuild Tests	Log	dotnet/sdk#51727
1199032	dotnet/sdk	Run TestBuild Tests	Log	dotnet/sdk#51535
1203612	dotnet/sdk	🟣 Run FullFramework Tests	Log	dotnet/sdk#51727

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
1	21	79

[marcpopMSFT]

@tmat we still see this fairly frequently. Any luck? Here's a recent one from a codeflow PR: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-sdk-refs-pull-41616-merge-58511284b77c40778a/dotnet-watch.Tests.dll.1/1/console.418de7ca.log?helixlogtype=result

[tmat]

No luck yet. This might help: https://github.com/dotnet/sdk/blob/d45e40d665e647804de03d2b4eec2aae9c488cb5/test/dotnet-watch.Tests/xunit.runner.json

It will print out test info every 20 seconds a test is running.

[marcpopMSFT]

@tmat another failure this morning shows a bunch of build processes for watch apps still running... https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-sdk-refs-pull-41958-merge-84357342ee154a5088/dotnet-watch.Tests.dll.2/1/console.3dddc463.log?helixlogtype=result

[tmat]

Looks like that's in 8.0 branch. Porting test diagnostics from 9: https://github.com/dotnet/sdk/pull/41963

[Forgind]

This has been making one of my PRs fail today, so I looked into it a bit. Based on when this issue was created and which tests are failing, I suspect this PR: https://github.com/dotnet/sdk/pull/39618

I asked jasonmalinowski whether the MSBuildWorkspace is supposed to support F# because I'd found a code snippet that suggested that it shouldn't, and he said it doesn't and never did. Did that PR make us use MSBuildWorkspace in a different way than we had before?

[Forgind]

The changes to HotReloadDotNetWatcher look most suspicious to me. It looks like a substantial change to how we call MSBuild.

[Forgind]

Ping @tmat This is still a serious pain point, currently preventing MSBuild from merging their code flow to 9.0.2xx. I already clicked rerun twice. I pointed to a PR that I think is likely to blame.

[tmat]

Fixes targeting 9.0.3xx: https://github.com/dotnet/sdk/pull/45999, https://github.com/dotnet/sdk/pull/46141

[nagilson]

@tmat Thank you for making some PRs to try to remedy this -- we are still seeing this in 9.0.2xx, 9.0.1xx, and main, 10 preview 2. Do your changes need to be ported?

[tmat]

They don't flow automatically to main?

Re 9.0.2xx, 9.0.1xx - these are product changes. Do they meet the servicing bar?

[nagilson]

They should flow to main, but either something went wrong and it didn't for those PRs, or the fix did not work, since it is still showing up in .net 10 preview 2: https://github.com/dotnet/sdk/pull/46983

I dont think they would meet the bar that, so good point. 9.0.3xx though, they probably do.

[tmat]

Might be something else causing failure in 10 P2. Do you have a link to the failed run? The PR is merged.

[baronfel]

This has spiked a lot in the last week - 83 hits out of ~140 for the month. @tmat would you be able to dig in and see if there's something fundamental that needs addressing?

[Forgind]

Did you ever look into https://github.com/dotnet/sdk/issues/40006#issuecomment-2311025564?

[tmat]

I can't access logs from the linked PRs:

[tmat]

@Forgind Yes, there is no issue with F#.

[tmat]

If anyone has a PR with dotnet-watch failure, please download the test logs and share them here. Helix links are not useful.