runtime icon indicating copy to clipboard operation
runtime copied to clipboard
verified publisher icon dotnet

CI flakiness: mono interpreter build getting killed

Open lewing opened this issue 8 months ago • 4 comments

Build Information

Build: https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=998900&view=logs&j=3e93a48f-6568-53f2-76b3-1285c183c39d Build error leg or test failing: Build / browser-wasm linux Release AllSubsets_Mono_RuntimeTests monointerpreter / Build Tests Pull request: https://github.com/dotnet/runtime/pull/114058

Error Message

Fill the error message using step by step known issues guidance.

{
  "ErrorMessage": "",
  "ErrorPattern": "RuntimeVariant=monointerpreter.*exited with code 137",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=998900 Error message validated: [RuntimeVariant=monointerpreter*exited with code 137.] Result validation: :x: Known issue did not match with the provided build. Validation performed at: 4/1/2025 9:14:37 PM UTC

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=998900 Error message validated: [RuntimeVariant=monointerpreter.*exited with code 137] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 4/1/2025 9:15:42 PM UTC

Report

Build Definition Step Name Console log Pull Request
1067996 dotnet/runtime Build Tests Log dotnet/runtime#113956
1067903 dotnet/runtime Build Tests Log dotnet/runtime#116626
1067646 dotnet/runtime Build Tests Log dotnet/runtime#115916
1067636 dotnet/runtime Build Tests Log dotnet/runtime#116619
1067411 dotnet/runtime Build Tests Log dotnet/runtime#116626
1066860 dotnet/runtime Build Tests Log dotnet/runtime#116082
1066684 dotnet/runtime Build Tests Log dotnet/runtime#116512
1066232 dotnet/runtime Build Tests Log dotnet/runtime#115393
1066121 dotnet/runtime Build Tests Log dotnet/runtime#116525
1065389 dotnet/runtime Build Tests Log
1065070 dotnet/runtime Build Tests Log dotnet/runtime#116383
1064009 dotnet/runtime Build Tests Log dotnet/runtime#116459
1063870 dotnet/runtime Build Tests Log
1063280 dotnet/runtime Build Tests Log dotnet/runtime#116459
1063010 dotnet/runtime Build Tests Log
1062955 dotnet/runtime Build Tests Log dotnet/runtime#115576
1062515 dotnet/runtime Build Tests Log dotnet/runtime#115537
1062485 dotnet/runtime Build Tests Log
1062356 dotnet/runtime Build Tests Log dotnet/runtime#116428
1062280 dotnet/runtime Build Tests Log dotnet/runtime#116384
1061978 dotnet/runtime Build Tests Log dotnet/runtime#116411
1061885 dotnet/runtime Build Tests Log
1061710 dotnet/runtime Build Tests Log
1061388 dotnet/runtime Build Tests Log dotnet/runtime#116357
1059782 dotnet/runtime Build Tests Log dotnet/runtime#116095
1059458 dotnet/runtime Build Tests Log dotnet/runtime#116326
1059334 dotnet/runtime Build Tests Log dotnet/runtime#116082
1058553 dotnet/runtime Build Tests Log dotnet/runtime#114682
1057873 dotnet/runtime Build Tests Log
1057689 dotnet/runtime Build Tests Log dotnet/runtime#115513
1057662 dotnet/runtime Build Tests Log dotnet/runtime#115774
1057638 dotnet/runtime Build Tests Log
1057438 dotnet/runtime Build Tests Log dotnet/runtime#115855
1057424 dotnet/runtime Build Tests Log dotnet/runtime#116145
1057101 dotnet/runtime Build Tests Log
1056949 dotnet/runtime Build Tests Log
1056708 dotnet/runtime Build Tests Log dotnet/runtime#115513
1055500 dotnet/runtime Build Tests Log dotnet/runtime#116166
1054323 dotnet/runtime Build Tests Log dotnet/runtime#116107
1054044 dotnet/runtime Build Tests Log dotnet/runtime#115609
1053821 dotnet/runtime Build Tests Log dotnet/runtime#113956
1051675 dotnet/runtime Build Tests Log
1050749 dotnet/runtime Build Tests Log
1050568 dotnet/runtime Build Tests Log dotnet/runtime#115704
1049991 dotnet/runtime Build Tests Log dotnet/runtime#115858
1049202 dotnet/runtime Build Tests Log
1047881 dotnet/runtime Build Tests Log
1047361 dotnet/runtime Build Tests Log dotnet/runtime#115818
1045781 dotnet/runtime Build Tests Log dotnet/runtime#115750
1045198 dotnet/runtime Build Tests Log dotnet/runtime#115724
1045019 dotnet/runtime Build Tests Log dotnet/runtime#114148
1044192 dotnet/runtime Build Tests Log dotnet/runtime#115544
1043795 dotnet/runtime Build Tests Log dotnet/runtime#115576
1043787 dotnet/runtime Build Tests Log
1043474 dotnet/runtime Build Tests Log dotnet/runtime#115039
1043408 dotnet/runtime Build Tests Log dotnet/runtime#115531
1043376 dotnet/runtime Build Tests Log dotnet/runtime#115576
1043385 dotnet/runtime Build Tests Log
1043365 dotnet/runtime Build Tests Log
1043357 dotnet/runtime Build Tests Log dotnet/runtime#115545
1043242 dotnet/runtime Build Tests Log dotnet/runtime#115408
1043079 dotnet/runtime Build Tests Log dotnet/runtime#113956
1042975 dotnet/runtime Build Tests Log dotnet/runtime#115574
1042922 dotnet/runtime Build Tests Log dotnet/runtime#115591
1042899 dotnet/runtime Build Tests Log
1042786 dotnet/runtime Build Tests Log dotnet/runtime#115574
1042728 dotnet/runtime Build Tests Log dotnet/runtime#115549
1042673 dotnet/runtime Build Tests Log dotnet/runtime#114438
1042459 dotnet/runtime Build Tests Log dotnet/runtime#115573
1042605 dotnet/runtime Build Tests Log dotnet/runtime#115058
1042420 dotnet/runtime Build Tests Log dotnet/runtime#115563
1042578 dotnet/runtime Build Tests Log dotnet/runtime#115574
1042552 dotnet/runtime Build Tests Log dotnet/runtime#115610
1042330 dotnet/runtime Build Tests Log dotnet/runtime#115408
1042465 dotnet/runtime Build Tests Log dotnet/runtime#115605
1042414 dotnet/runtime Build Tests Log dotnet/runtime#115427
1042349 dotnet/runtime Build Tests Log dotnet/runtime#115573
1041448 dotnet/runtime Build Tests Log dotnet/runtime#115553
1042324 dotnet/runtime Build Tests Log dotnet/runtime#113956
1041806 dotnet/runtime Build Tests Log dotnet/runtime#115494
1042282 dotnet/runtime Build Tests Log dotnet/runtime#115251

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
5 23 81

lewing avatar Apr 01 '25 21:04 lewing

Tagging subscribers to this area: @akoeplinger, @matouskozak See info in area-owners.md if you want to be subscribed.

dotnet-policy-service[bot] avatar Apr 01 '25 21:04 dotnet-policy-service[bot]

fyi: @BrzVlad

matouskozak avatar Apr 02 '25 07:04 matouskozak

We use mono interpreter to run msbuild, right ?

This is building 3 groups of many small tests

Image

That combined with WASM toolchain memory demands makes msbuild run out of memory (docker kills it). This is OOM of the msbuild, not OOM of the individual test projects. I think this is OOM kill, not timeout because the duration before the kill fluctuates.

As far as I can tell, it never passes for WASM and it's killed in the group 2 already.

Log

2025-05-15T23:09:37.6650337Z   Number of test projects in group 2: 2366
2025-05-15T23:09:37.7341670Z   Building managed test group 2: "/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"
2025-05-15T23:09:37.9271786Z   MSBuild version 17.14.0-preview-25177-05+903614e16 for .NET
2025-05-15T23:09:37.9852691Z   
2025-05-15T23:09:40.5433817Z     Number of test projects in group 2: 2366
2025-05-15T23:09:37.7341670Z   Building managed test group 2: "/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"
2025-05-15T23:09:37.9271786Z   MSBuild version 17.14.0-preview-25177-05+903614e16 for .NET
2025-05-15T23:09:37.9852691Z   
2025-05-15T23:09:40.5433817Z     Number of test projects in group 2: 2366
2025-05-15T23:09:41.8292504Z     XUnitWrapperGenerator -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/Common/XUnitWrapperGenerator/XUnitWrapperGenerator/XUnitWrapperGenerator.dll
2025-05-15T23:09:42.6596953Z     KeepAliveBoxOpt -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.dll
2025-05-15T23:09:42.9728086Z     sinlib_cs -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/Boxing/xlang/sinlib_cs/sinlib_cs.dll
2025-05-15T23:09:43.2633097Z     testlib_misc -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/misc/testlib_misc/testlib_misc.dll
2025-05-15T23:09:43.6004947Z     threads1_cs_do -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/misc/threads1_cs_do/threads1_cs_do.dll
2025-05-15T23:09:43.8861163Z     testlib_xassem -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Methodical/cctor/xassem/testlib_xassem/testlib_xassem.dll
...
2025-05-15T23:23:01.9327210Z     b53977 -> /__w/1/s/artifacts/tests/coreclr/browser.wasm.Release/JIT/Regression/CLR-x86-JIT/V1-M12-Beta2/b53977/b53977/b53977.dll
2025-05-15T23:23:08.5186861Z   /__w/1/s/dotnet.sh: line 27: 50331 Killed                  ${dotnetPath} "$@"
2025-05-15T23:23:08.5814254Z ##[error]src/tests/build.proj(501,5): error MSB3073: The command ""/__w/1/s/dotnet.sh" msbuild /__w/1/s/src/tests/build.proj /t:Build "/p:TargetArchitecture=wasm" "/p:Configuration=Release" "/p:LibrariesConfiguration=Release" "/p:TasksConfiguration=Release" "/p:TargetOS=browser" "/p:ToolsOS=" "/p:PackageOS=" "/p:RuntimeFlavor=mono" "/p:RuntimeVariant=monointerpreter" "/p:CLRTestBuildAllTargets=" "/p:UseCodeFlowEnforcement=" "/p:__TestGroupToBuild=2" "/p:__SkipRestorePackages=1" /nodeReuse:false /m:1 /bl:/__w/1/s/artifacts//log/Release/InnerManagedTestBuild.2.binlog /p:ContinuousIntegrationBuild=true "/p:DevTeamProvisioning=-"" exited with code 137.

pavelsavara avatar May 16 '25 07:05 pavelsavara

I disabled part of the test on browser, because 100% fail rate was not helping anybody. This is ActiveIssue now.

pavelsavara avatar May 16 '25 09:05 pavelsavara

removing blocking-clean-ci as it has not failed in 30 days

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

jeffschwMSFT avatar Sep 19 '25 17:09 jeffschwMSFT