runtime icon indicating copy to clipboard operation
runtime copied to clipboard

[8.0] Timeout - Wasm.Build.Tests.WasmTemplateTests.RunWithDifferentAppBundleLocations

Open jozkee opened this issue 6 months ago • 9 comments

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=1069267 Build error leg or test failing: Workloads-NoFingerprint-ST-Wasm.Build.Tests.WorkItemExecution Pull request: N/A

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": ["[Long Running Test] 'Wasm.Build.Tests.WasmTemplateTests.RunWithDifferentAppBundleLocations'", " exited with code 137"],
  "ErrorPattern": "",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1069267 Error message validated: [[Long Running Test] 'Wasm.Build.Tests.WasmTemplateTests.RunWithDifferentAppBundleLocations'] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 6/16/2025 10:36:07 PM UTC

Report

Build Definition Test Pull Request
1068959 dotnet/runtime Workloads-NoFingerprint-ST-Wasm.Build.Tests.WorkItemExecution dotnet/runtime#116685
1069267 dotnet/runtime Workloads-NoFingerprint-ST-Wasm.Build.Tests.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
2 2 2

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=1069267 Error message validated: [[Long Running Test] 'Wasm.Build.Tests.WasmTemplateTests.RunWithDifferentAppBundleLocations' exited with code 137] Result validation: :x: Known issue did not match with the provided build. Validation performed at: 6/17/2025 3:45:26 PM UTC

Report

Build Definition Test Pull Request
1068959 dotnet/runtime Workloads-NoFingerprint-ST-Wasm.Build.Tests.WorkItemExecution dotnet/runtime#116685
1069267 dotnet/runtime Workloads-NoFingerprint-ST-Wasm.Build.Tests.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
2 2 2

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

jozkee avatar Jun 16 '25 22:06 jozkee

The pattern here isn't related to the error at all so any failure in this test will match this issue. The actual problem is the vm is running out of memory and killing the process.

lewing avatar Jun 17 '25 15:06 lewing

The actual problem is the vm is running out of memory and killing the process.

We have plenty of cases where OOM is reported this way https://github.com/dotnet/runtime/issues/113686 https://github.com/dotnet/runtime/issues/111812 https://github.com/dotnet/runtime/issues/115543 https://github.com/dotnet/runtime/issues/115683

and sometimes addressed: https://github.com/dotnet/runtime/pull/116460

jozkee avatar Jun 17 '25 15:06 jozkee

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Tagging subscribers to this area: @akoeplinger, @matouskozak, @simonrozsival See info in area-owners.md if you want to be subscribed.

The actual problem is the vm is running out of memory and killing the process.

We have plenty of cases where OOM is reported this way #113686 #111812 #115543 #115683

and sometimes addressed: #116460

I wasn't suggesting we don't want to fix them where possible, just that making the pattern more precise is generally better.

lewing avatar Jun 17 '25 15:06 lewing

I've added a check for the exit code here so the pattern will only match when the container kills the process.

lewing avatar Jun 17 '25 15:06 lewing

removing blocking-clean-ci as it has not failed in 30 days

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

jeffschwMSFT avatar Jul 17 '25 16:07 jeffschwMSFT

Hit in https://github.com/dotnet/runtime/pull/118763, changing title to 8.0 and reopening

jozkee avatar Aug 15 '25 17:08 jozkee

Note that we might be underreporting the number of failures. Sometimes the browser-wasm linux Release WasmBuildTests itself times out before the test times out so a generic test failure shows up: https://github.com/dotnet/runtime/pull/120781/checks?check_run_id=52882999404. If you manually dig through the helix logs/json/links it will lead to this test running for 90ish minutes and timing out.

PranavSenthilnathan avatar Oct 16 '25 21:10 PranavSenthilnathan