dnceng icon indicating copy to clipboard operation
dnceng copied to clipboard

The active test run was aborted. Reason: Test host process crashed

Open fanyang-mono opened this issue 2 years ago • 8 comments
trafficstars

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=351450

Build leg reported

chrome-DebuggerTests.EvaluateOnCallFrameTests.WorkItemExecution

Pull Request

https://github.com/dotnet/runtime/pull/89217

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "The active test run was aborted. Reason: Test host process crashed",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

  • [ ] Feature changes/additions
  • [ ] Bug fixes
  • [ ] Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Report

Build Definition Step Name Console log Pull Request
1044450 dotnet/roslyn Run Integration Tests Log
1044091 dotnet/roslyn Run Integration Tests Log
1043013 dotnet/roslyn Run Integration Tests Log
1041627 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78357
1041253 dotnet/roslyn Run Integration Tests Log
1040317 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78562
1040236 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78558
1039874 dotnet/roslyn Run Integration Tests Log
1039725 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78388
1039277 dotnet/roslyn Run Integration Tests Log
1038918 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78357
1038511 dotnet/roslyn Run Integration Tests Log
1038478 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78516
1038318 dotnet/roslyn Run Integration Tests Log
1038268 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78542
1037917 dotnet/roslyn Run Unit Tests Log
1037926 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78522
1037925 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78525
1037876 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78532
1037397 dotnet/roslyn Run Integration Tests Log
1037307 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78507
1037256 dotnet/roslyn Run Integration Tests Log
1037250 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78514
1037240 dotnet/roslyn Run Integration Tests Log
1037078 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78511
1036839 dotnet/roslyn Run Integration Tests Log
1036720 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78498
1036403 dotnet/roslyn Run Integration Tests Log
1036285 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78498
1035007 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78140
1034620 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78458
1034237 dotnet/roslyn Run Integration Tests Log
1034065 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#76691
1033010 dotnet/roslyn Run Integration Tests Log
1032686 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78420
1032636 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78401
1032417 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78401
1031089 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78357
1031921 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#77088
1030021 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78376
1029927 dotnet/roslyn Run Integration Tests Log
1029925 dotnet/roslyn Run Integration Tests Log
1029907 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78369
1029899 dotnet/roslyn Run Integration Tests Log
1028890 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78357
2697948 dotnet-fsharp CmdLine@2 Log
1029076 dotnet/roslyn Run Integration Tests Log
2697752 dotnet-fsharp CmdLine@2 Log
1028734 dotnet/roslyn Run Integration Tests Log
1028647 dotnet/roslyn Run Integration Tests Log
2696596 dotnet-fsharp CmdLine Log
1027081 dotnet/roslyn Run Integration Tests Log
1026935 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78220
1024848 dotnet/roslyn Run Integration Tests Log
1024264 dotnet/roslyn Run Integration Tests Log
1024242 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78213
1024156 dotnet/roslyn Run Integration Tests Log
1023228 dotnet/roslyn Run Integration Tests Log
1022065 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78220
1020803 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78205
1020781 dotnet/roslyn Run Integration Tests Log dotnet/roslyn#78208
Build Definition Test Pull Request
1044609 dotnet/sdk Microsoft.WebTools.AspireService.Tests.dll.WorkItemExecution dotnet/sdk#49040
2711186 dotnet-sdk Microsoft.WebTools.AspireService.Tests.dll.WorkItemExecution #50157
1043895 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution dotnet/aspnetcore#61929
1043600 dotnet/sdk dotnet-watch.Tests.dll.5.WorkItemExecution dotnet/sdk#49014
1043549 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution dotnet/aspnetcore#61964
1043554 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48919
1043415 dotnet/sdk dotnet-watch.Tests.dll.5.WorkItemExecution dotnet/sdk#48843
1042381 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution dotnet/aspnetcore#61633
1042570 dotnet/sdk Microsoft.DotNet.Watch.UnitTests.ApplyDeltaTests.Microsoft.DotNet.Watch.UnitTests.ApplyDeltaTests.Aspire dotnet/sdk#48982
1042686 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48984
1042541 dotnet/sdk Microsoft.WebTools.AspireService.Tests.dll.WorkItemExecution dotnet/sdk#48929
1042282 dotnet/runtime HostActivation.Tests.WorkItemExecution dotnet/runtime#115251
1041972 dotnet/roslyn workitem_7.WorkItemExecution dotnet/roslyn#78582
1041964 dotnet/sdk dotnet-watch.Tests.dll.4.WorkItemExecution dotnet/sdk#48966
1041088 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48925
1041049 dotnet/sdk Microsoft.WebTools.AspireService.Tests.dll.WorkItemExecution dotnet/sdk#48938
1040634 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48909
1040298 dotnet/sdk dotnet-watch.Tests.dll.5.WorkItemExecution dotnet/sdk#48910
1040066 dotnet/aspnetcore Microsoft.AspNetCore.Server.HttpSys.RequestTests.Request_MultiplePrefixes
1039881 dotnet/sdk Microsoft.NET.Sdk.Razor.Tool.Tests.dll.1.WorkItemExecution dotnet/sdk#48882
1039826 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48907
1039331 dotnet/sdk Microsoft.WebTools.AspireService.Tests.dll.WorkItemExecution dotnet/sdk#48813
1038537 dotnet/aspnetcore Microsoft.AspNetCore.Components.Endpoints.Tests--net8.0.WorkItemExecution
2706287 dotnet-roslyn workitem_204.WorkItemExecution
1037623 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48883
1037210 dotnet/roslyn workitem_6.WorkItemExecution dotnet/roslyn#78513
2704797 dotnet-aspnetcore Microsoft.AspNetCore.Http.Results.Tests--net8.0.WorkItemExecution
1036710 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48737
1036711 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48751
1036716 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48859
1035704 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48697
1035154 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48609
1035010 dotnet/aspnetcore Microsoft.AspNetCore.Server.HttpSys.Listener.ResponseHeaderTests.AddingControlCharactersToHeadersThrows
1034754 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48697
1034405 dotnet/roslyn workitem_9.WorkItemExecution dotnet/roslyn#78436
1033821 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48697
1033729 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48813
1032667 dotnet/roslyn workitem_6.WorkItemExecution dotnet/roslyn#78424
1031911 dotnet/sdk dotnet-watch.Tests.dll.4.WorkItemExecution dotnet/sdk#48752
1030360 dotnet/aspnetcore Microsoft.AspNetCore.Http.Abstractions.Tests--net10.0.WorkItemExecution dotnet/aspnetcore#61745
1031447 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution
1030340 dotnet/sdk Microsoft.NET.Sdk.Razor.Tool.Tests.dll.1.WorkItemExecution dotnet/sdk#48777
1029908 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48773
2697816 dotnet-aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution
2697785 dotnet-roslyn workitem_203.WorkItemExecution
1027246 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48735
1027155 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48731
1027076 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48690
1026964 dotnet/sdk Microsoft.NET.Sdk.Razor.Tool.Tests.dll.1.WorkItemExecution dotnet/sdk#48725
1026526 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution
1025043 dotnet/sdk Microsoft.WebTools.AspireService.Tests.dll.WorkItemExecution dotnet/sdk#48689
1024954 dotnet/sdk Microsoft.WebTools.AspireService.Tests.dll.WorkItemExecution dotnet/sdk#47076
2693348 dotnet-aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution
1023300 dotnet/sdk Microsoft.WebTools.AspireService.Tests.dll.WorkItemExecution dotnet/sdk#48640
1023289 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48611
1023164 dotnet/aspnetcore Microsoft.AspNetCore.Mvc.FunctionalTests--net10.0.WorkItemExecution dotnet/aspnetcore#61503
1023178 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48587
1023168 dotnet/sdk Microsoft.NET.Sdk.Razor.Tool.Tests.dll.1.WorkItemExecution dotnet/sdk#48637
1021108 dotnet/sdk Microsoft.DotNet.Watch.UnitTests.ApplyDeltaTests.Microsoft.DotNet.Watch.UnitTests.ApplyDeltaTests.Aspire dotnet/sdk#48028
1020922 dotnet/sdk dotnet-watch.Tests.dll.1.WorkItemExecution dotnet/sdk#48589
1020766 dotnet/aspnetcore Templates.Mvc.Test.BlazorTemplateTest.BlazorWebTemplate_IndividualAuth(args: ["--interactivity Auto"]) dotnet/aspnetcore#61554
1020770 dotnet/roslyn workitem_273.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
2 32 123

Known issue validation

Build: :mag_right: Result validation: :warning: Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: :mag_right:" line. Validation performed at: 2/8/2024 9:24:30 PM UTC

fanyang-mono avatar Jul 25 '23 17:07 fanyang-mono

Hello @fanyang-mono, could you please update the "ErrorMessage" : "" by following the step by step documentation on how to create a known issue

andriipatsula avatar Jul 26 '23 08:07 andriipatsula

Updated.

fanyang-mono avatar Jul 26 '23 14:07 fanyang-mono

This is duplicate of https://github.com/dotnet/runtime/issues/89409, except here it's not only wasm debugger

pavelsavara avatar Nov 07 '23 10:11 pavelsavara

@pavelsavara and @fanyang-mono, should this issue be closed in favour of dotnet/runtime#94560❔ if not, the error message isn't sufficiently precise to understand what (if anything) we (DNCEng) could do about it.

dougbu avatar Nov 23 '23 00:11 dougbu

@pavelsavara and @fanyang-mono, should this issue be closed in favour of

You mean https://github.com/dotnet/runtime/issues/89409

I think we should keep this one, because this one has some hits which are not just wasm debugger related. Here we see other repo failures, on roslyn and aspnetcore CI.

if not, the error message isn't sufficiently precise to understand what (if anything) we (DNCEng) could do about it.

I have not investigated it myself yet. But got questions which could help to narrow it down

  • are there crash dump files on the machine ?
  • are there some hints on what's going on in the log files before crash ?
  • is this OOM or stack overflow in disguise ?
  • did we run out of disk space or number of inodes ?
  • hit some limit in docker container ?
  • is this always the same OS, version, image ?
  • could this be hardware issue ?

pavelsavara avatar Nov 23 '23 08:11 pavelsavara

This issue was originally created to track issues happened to wasm debugger tests (belonging to dotnet/runtime repo). However, the error message was generic enough to include issues from other repos, such as dotnet/roslyn and dotnet/aspnetcore. We need insights from owners from dotnet/roslyn and dotnet/aspnetcore to confirm that they are aware of these issue. And if they are okay to close this issue as well. We could close then.

fanyang-mono avatar Nov 28 '23 15:11 fanyang-mono

We just got an instance of this in the dotnet/aspire repo. It wasn't a WASM related failure for us. Just looks like the test process crashed and took a few tests with it.

mitchdenny avatar Jun 26 '24 22:06 mitchdenny

@marcpopMSFT was working on getting the test infrastructure to generate crash dumps when it times out to help with investigation. I think it was generating the dumps but not yet correctly uploading them to the artifacts.

We are waiting to hear from the infra team on an issue with this.

nagilson avatar Dec 03 '24 20:12 nagilson