runtime icon indicating copy to clipboard operation
runtime copied to clipboard

Published 20 hours ago • verified publisher icon dotnet

Test failure JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

Open VincentBu opened this issue 3 years ago • 17 comments

Run：runtime-coreclr gcstress-extra 20220508.1

Failed test:

coreclr OSX arm64 Checked gcstress0xc_jitstress2 @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

Error message:

cmdLine:/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 5/8/2022 5:10:55 PM, end: 5/8/2022 6:40:58 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/B4220957/w/A7D308F6/uploads/Arrays/lcs/lcs2_r/output.txt
Raw output:
SKIPPING EXECUTION BECAUSE COMPlus_GCStress IS SET

cmdLine:/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 5/8/2022 5:10:55 PM, end: 5/8/2022 6:40:58 PM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh
Expected: True
Actual:   False


Stack trace
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)
Queued OS Arch Pipeline
2022-05-09T00:03:15.936Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2022-04-09T08:49:58.428Z osx.1015.amd64.open x64 runtime-staging Release
2022-03-30T20:41:58.472Z osx.1015.amd64.open x64 runtime-staging Release
2022-03-16T07:23:50.554Z ubuntu.1804.amd64.android.29.open.svc x64 runtime-extra-platforms Release

VincentBu avatar May 10 '22 01:05 VincentBu

Tagging subscribers to this area: @JulieLeeMSFT See info in area-owners.md if you want to be subscribed.

Issue Details

Run：runtime-coreclr gcstress-extra 20220508.1

Failed test:

coreclr OSX arm64 Checked gcstress0xc_jitstress2 @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

Error message:

cmdLine:/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 5/8/2022 5:10:55 PM, end: 5/8/2022 6:40:58 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/B4220957/w/A7D308F6/uploads/Arrays/lcs/lcs2_r/output.txt
Raw output:
SKIPPING EXECUTION BECAUSE COMPlus_GCStress IS SET

cmdLine:/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 5/8/2022 5:10:55 PM, end: 5/8/2022 6:40:58 PM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh
Expected: True
Actual:   False


Stack trace
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)
Queued OS Arch Pipeline
2022-05-09T00:03:15.936Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2022-04-09T08:49:58.428Z osx.1015.amd64.open x64 runtime-staging Release
2022-03-30T20:41:58.472Z osx.1015.amd64.open x64 runtime-staging Release
2022-03-16T07:23:50.554Z ubuntu.1804.amd64.android.29.open.svc x64 runtime-extra-platforms Release
Author: VincentBu
Assignees: -
Labels:

arch-arm64, os-mac-os-x, GCStress, JitStress, area-CodeGen-coreclr, untriaged, blocking-clean-ci-optional
Milestone: -

msftbot[bot] avatar May 10 '22 01:05 msftbot[bot]

@jakobbotsch - assigning to you since you have osx.

kunalspathak avatar May 19 '22 05:05 kunalspathak

This looks like another occurrence of #68884. The test is disabled in GC stress, gets skipped, but the OutOfProcess execution mechanism seems to hang after that. cc @trylek, it seems like there is some form of intermittent deadlock in the OutOfProcess execution mechanism, do you have any idea how we can get to the bottom of it?

jakobbotsch avatar May 20 '22 09:05 jakobbotsch

Tagging subscribers to this area: @hoyosjs See info in area-owners.md if you want to be subscribed.

Issue Details

Run：runtime-coreclr gcstress-extra 20220508.1

Failed test:

coreclr OSX arm64 Checked gcstress0xc_jitstress2 @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

Error message:

cmdLine:/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 5/8/2022 5:10:55 PM, end: 5/8/2022 6:40:58 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/B4220957/w/A7D308F6/uploads/Arrays/lcs/lcs2_r/output.txt
Raw output:
SKIPPING EXECUTION BECAUSE COMPlus_GCStress IS SET

cmdLine:/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 5/8/2022 5:10:55 PM, end: 5/8/2022 6:40:58 PM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh
Expected: True
Actual:   False


Stack trace
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)
Queued OS Arch Pipeline
2022-05-09T00:03:15.936Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2022-04-09T08:49:58.428Z osx.1015.amd64.open x64 runtime-staging Release
2022-03-30T20:41:58.472Z osx.1015.amd64.open x64 runtime-staging Release
2022-03-16T07:23:50.554Z ubuntu.1804.amd64.android.29.open.svc x64 runtime-extra-platforms Release
Author: VincentBu
Assignees: jakobbotsch
Labels:

arch-arm64, os-mac-os-x, GCStress, JitStress, area-Infrastructure-coreclr, blocking-clean-ci-optional
Milestone: 7.0.0

msftbot[bot] avatar May 20 '22 09:05 msftbot[bot]

I managed to get a repro by running CoreclrTestWrapperLib.RunTest in a loop with COMPlus_GCStress=c and COMPlus_JitStress=2 on macOS arm64. Trying to track down now what the problem is.

jakobbotsch avatar May 20 '22 11:05 jakobbotsch

@dotnet/area-system-diagnostics-process Can you please confirm that the following code looks correct: https://github.com/dotnet/runtime/blob/8d4a724883715cc5ae71e020b22762136f7ce50b/src/tests/Common/Coreclr.TestWrapper/CoreclrTestWrapperLib.cs#L306-L334

On macOS arm64 I am frequently hitting a hang on the WaitForExit call when I run this code in a loop under COMPlus_GCStress=c. The child process is bash that runs a test wrapper script that prints a message then immediately exits.

Note that GCStress introduces a lot of execution time variability. I suspect that is part of the problem.

Stack traces from all threads during the hang are available here.

Here is a test project and script to reproduce the problem. repro.zip To use:

  1. Build the project in release mode
  2. Build checked runtime with release libs
  3. Make sure to cd into the dir so that repro.sh is in current dir
  4. COMPlus_GCStress=c corerun <output dll>

If the last step does not repro the hang within the first 100 iterations then ctrl-C and try step 4 again.

jakobbotsch avatar May 20 '22 12:05 jakobbotsch

If I change the copy tasks to:

                Task copyOutput = Task.Run(async () =>
                {
                    try
                    {
                        Console.WriteLine($"  [{index}] Starting CopyTo output");
                        await process.StandardOutput.BaseStream.CopyToAsync(outputStream, 4096, cts.Token);
                        Console.WriteLine($"  [{index}] Done with CopyTo output");
                    }
                    catch (Exception ex) when (ex is not OperationCanceledException)
                    {
                        Console.WriteLine(ex);
                    }
                });
                Task copyError = Task.Run(async () =>
                {
                    try
                    {
                        Console.WriteLine($"  [{index}] Starting CopyTo err");
                        await process.StandardError.BaseStream.CopyToAsync(errorStream, 4096, cts.Token);
                        Console.WriteLine($"  [{index}] Done with CopyTo err");
                    }
                    catch (Exception ex) when (ex is not OperationCanceledException)
                    {
                        Console.WriteLine(ex);
                    }
                });

Then on hang I see:

[2] Running
  [2] WaitForExit(600000) on PID 73072
  [2] Starting CopyTo err
  [2] Starting CopyTo output
  [2] Done with CopyTo err
  [2] Done with CopyTo output

The bash process is still alive at this point.

jakobbotsch avatar May 20 '22 12:05 jakobbotsch

Honestly I don't think that marking these tests as out-of-proc actually fixed the hangs we were seeing, it just made them much less frequent. My working theory based on Andrew's initial analysis of the problem is that the GC stress logic can sometimes cause some form of starvation w.r.t. draining the finalization queue. Apparently the problem was much more pronounced when running the tests in-proc, according to my working theory this is likely because the one bigger process comprising hundreds of test cases is naturally much heavier with regard to GC allocation compared to running just a single test case as a separate process. It is great that you managed to shed more light on what's happening as marking the tests out-of-proc was just a mitigation measure, not the real solution for the underlying problem.

trylek avatar May 22 '22 18:05 trylek

What I said above about the bash process still being alive is not totally right. The bash process shows up as (bash) in ps. From my googling, the parentheses seem to indicate that something might have gone wrong with an execve syscall, but I am not totally sure. I am also not able to attach lldb to the (bash) process, so it might be that Process.Start actually failed somewhere.

@dotnet/area-system-diagnostics-process Can one of you please investigate this hang further? It might potentially be a GCStress issue, although it seems to happen on macOS arm64 only. It would be good to pin down the events that happen in Process.Start and Process.WaitForExit.

jakobbotsch avatar May 23 '22 09:05 jakobbotsch

Tagging subscribers to this area: @dotnet/area-system-diagnostics-process See info in area-owners.md if you want to be subscribed.

Issue Details

Run：runtime-coreclr gcstress-extra 20220508.1

Failed test:

coreclr OSX arm64 Checked gcstress0xc_jitstress2 @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

Error message:

cmdLine:/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 5/8/2022 5:10:55 PM, end: 5/8/2022 6:40:58 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/B4220957/w/A7D308F6/uploads/Arrays/lcs/lcs2_r/output.txt
Raw output:
SKIPPING EXECUTION BECAUSE COMPlus_GCStress IS SET

cmdLine:/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 5/8/2022 5:10:55 PM, end: 5/8/2022 6:40:58 PM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/B4220957/w/A7D308F6/e/JIT/Methodical/Methodical_r1/../Arrays/lcs/lcs2_r/lcs2_r.sh
Expected: True
Actual:   False


Stack trace
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)
Queued OS Arch Pipeline
2022-05-09T00:03:15.936Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2022-04-09T08:49:58.428Z osx.1015.amd64.open x64 runtime-staging Release
2022-03-30T20:41:58.472Z osx.1015.amd64.open x64 runtime-staging Release
2022-03-16T07:23:50.554Z ubuntu.1804.amd64.android.29.open.svc x64 runtime-extra-platforms Release
Author: VincentBu
Assignees: -
Labels:

arch-arm64, area-System.Diagnostics.Process, os-mac-os-x, GCStress, blocking-clean-ci-optional
Milestone: 7.0.0

msftbot[bot] avatar May 23 '22 09:05 msftbot[bot]

Failed again in: runtime-coreclr gcstress-extra 20220611.1

Failed test:

coreclr Linux arm Checked gcstress0xc_zapdisable_heapverify1 @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440

- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd
- JIT/Methodical/Arrays/lcs/lcs2_do/lcs2_do.cmd
- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd
- JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.cmd

coreclr Linux arm64 Checked gcstress0xc_zapdisable_heapverify1 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652

- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd
- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd
- JIT/Methodical/Arrays/lcs/lcs2_do/lcs2_do.cmd
- JIT/Methodical/Arrays/lcs/lcs2_d/lcs2_d.cmd
- JIT/Methodical/fp/exgen/10w5d_cs_r/10w5d_cs_r.cmd
- JIT/Methodical/fp/exgen/10w5d_cs_d/10w5d_cs_d.cmd
- JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.cmd

coreclr Linux arm64 Checked gcstress0xc_zapdisable_jitstress2 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652

- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd
- JIT/Methodical/Arrays/lcs/lcs2_d/lcs2_d.cmd
- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd
- JIT/Methodical/Arrays/lcs/lcs2_do/lcs2_do.cmd
- JIT/Methodical/fp/exgen/10w5d_cs_r/10w5d_cs_r.cmd
- JIT/Methodical/fp/exgen/10w5d_cs_d/10w5d_cs_d.cmd
- JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcs2_d/lcs2_d.cmd

coreclr OSX arm64 Checked gcstress0xc_jitstress2 @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

Error message:

Test Infrastructure Failure: System.IO.IOException: Unknown error 42424 : '/root/helix/work/workitem/uploads/Arrays/lcs/lcs2_r'
at System.IO.FileSystem.CreateDirectory(String fullPath)
at System.IO.Directory.CreateDirectory(String path)
at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
Expected: True
Actual:   False


Stack trace
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)
Queued OS Arch Column1
2022-06-12T08:37:37.57Z osx.1200.arm64.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2022-05-16T08:50:14.127Z osx.1200.arm64.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2022-05-09T00:03:15.936Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2

VincentBu avatar Jun 13 '22 01:06 VincentBu

Failed again in: runtime-coreclr gcstress-extra 20220703.1

Failed test:

coreclr Linux arm64 Checked gcstress0xc_zapdisable_heapverify1 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652

- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd
- JIT/Methodical/Arrays/lcs/lcs2_d/lcs2_d.cmd
- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd
- JIT/Methodical/Arrays/lcs/lcs2_do/lcs2_do.cmd
- JIT/Methodical/fp/exgen/10w5d_cs_r/10w5d_cs_r.cmd
- JIT/Methodical/fp/exgen/10w5d_cs_d/10w5d_cs_d.cmd
- JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.cmd

coreclr Linux arm64 Checked gcstress0xc_zapdisable_jitstress2 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652

- JIT/Methodical/Arrays/lcs/lcs2_d/lcs2_d.cmd
- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd
- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd
- JIT/Methodical/Arrays/lcs/lcs2_do/lcs2_do.cmd
- JIT/Methodical/fp/exgen/10w5d_cs_r/10w5d_cs_r.cmd
- JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.cmd
- JIT/Methodical/fp/exgen/10w5d_cs_d/10w5d_cs_d.cmd
- JIT/Methodical/doublearray/dblarray1_cs_r/dblarray1_cs_r.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable_heapverify1 @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcsvalbox_r/lcsvalbox_r.cmd

coreclr OSX arm64 Checked gcstress0xc_tailcallstress @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcs2_d/lcs2_d.cmd

Error message:

Test Infrastructure Failure: System.IO.IOException: Unknown error 42424 : '/root/helix/work/workitem/uploads/Arrays/lcs/lcs2_r'
at System.IO.FileSystem.CreateDirectory(String fullPath, UnixFileMode unixCreateMode)
at System.IO.Directory.CreateDirectory(String path)
at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
Expected: True
Actual:   False


Stack trace
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)

VincentBu avatar Jul 05 '22 02:07 VincentBu

Failed again in: runtime-coreclr gcstress0x3-gcstress0xc 20220717.1

Failed test:

coreclr OSX arm64 Checked gcstress0xc @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd

Error message:

cmdLine:/private/tmp/helix/working/B61D0A0F/w/B27509B2/e/JIT/Methodical/Methodical_ro/../Arrays/lcs/lcs2_ro/lcs2_ro.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 7/17/2022 8:25:33 AM, end: 7/17/2022 9:25:34 AM)

Return code:      -100
Raw output file:      /tmp/helix/working/B61D0A0F/w/B27509B2/uploads/Arrays/lcs/lcs2_ro/output.txt
Raw output:
SKIPPING EXECUTION BECAUSE COMPlus_GCStress IS SET

cmdLine:/private/tmp/helix/working/B61D0A0F/w/B27509B2/e/JIT/Methodical/Methodical_ro/../Arrays/lcs/lcs2_ro/lcs2_ro.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 7/17/2022 8:25:33 AM, end: 7/17/2022 9:25:34 AM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/B61D0A0F/w/B27509B2/e/JIT/Methodical/Methodical_ro/../Arrays/lcs/lcs2_ro/lcs2_ro.sh
Expected: True
Actual:   False


Stack trace
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)

VincentBu avatar Jul 18 '22 02:07 VincentBu

@dotnet/area-system-diagnostics-process, @jeffhandley this fails frequently and blocks gcstress test. Please take a look.

@dotnet/area-system-diagnostics-process Can one of you please investigate this hang further? It might potentially be a GCStress issue, although it seems to happen on macOS arm64 only. It would be good to pin down the events that happen in Process.Start and Process.WaitForExit.

JulieLeeMSFT avatar Jul 25 '22 16:07 JulieLeeMSFT

Failed again in : runtime-coreclr jitstress 20220724.3

Failed test:

coreclr OSX arm64 Checked gcstress0xc_zapdisable_jitstress2 @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcsvalbox_d/lcsvalbox_d.cmd
- JIT/Methodical/Arrays/lcs/lcsvalbox_r/lcsvalbox_r.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable_heapverify1 @ OSX.1200.ARM64.Open

- JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

Error message:

cmdLine:/private/tmp/helix/working/9B7B093A/w/B1150A05/e/JIT/Methodical/Methodical_d1/../Arrays/lcs/lcsvalbox_d/lcsvalbox_d.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 7/24/2022 4:29:54 PM, end: 7/24/2022 5:59:54 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/9B7B093A/w/B1150A05/uploads/Arrays/lcs/lcsvalbox_d/output.txt
Raw output:
SKIPPING EXECUTION BECAUSE COMPlus_GCStress IS SET

cmdLine:/private/tmp/helix/working/9B7B093A/w/B1150A05/e/JIT/Methodical/Methodical_d1/../Arrays/lcs/lcsvalbox_d/lcsvalbox_d.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 7/24/2022 4:29:54 PM, end: 7/24/2022 5:59:54 PM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/9B7B093A/w/B1150A05/e/JIT/Methodical/Methodical_d1/../Arrays/lcs/lcsvalbox_d/lcsvalbox_d.sh
Expected: True
Actual:   False


Stack trace
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)
Queued OS Arch Column1
2022-07-03T23:51:16.641Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable

VincentBu avatar Jul 26 '22 01:07 VincentBu

Failed again in: runtime-coreclr gcstress0x3-gcstress0xc 20220807.1

Failed test:

coreclr Linux arm Checked gcstress0x3 @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440

- ilverify/ILVerificationTests.sh

coreclr Linux arm64 Checked gcstress0x3 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20210531091519-97d8652

- JIT/Regression/JitBlue/Runtime_72845/Runtime_72845/Runtime_72845.sh

Error message:

cmdLine:/root/helix/work/workitem/e/ilverify/ILVerificationTests.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 8/7/2022 4:11:04 PM, end: 8/7/2022 5:11:05 PM)

Return code:      -100
Raw output file:      /root/helix/work/workitem/uploads/Reports/ilverify/ILVerificationTests.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false ILVerificationTests.dll ''

cmdLine:/root/helix/work/workitem/e/ilverify/ILVerificationTests.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 8/7/2022 4:11:04 PM, end: 8/7/2022 5:11:05 PM)
Test Harness Exitcode is : -100
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/e/ilverify/ILVerificationTests.sh
Expected: True
Actual:   False


Stack trace
   at ilverify._ILVerificationTests_._ILVerificationTests_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

v-wenyuxu avatar Aug 08 '22 01:08 v-wenyuxu

Queued OS Arch Column1
2022-08-07T14:40:21.014Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-07T09:14:58.202Z osx.1200.amd64.open x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2022-08-07T08:59:42.467Z osx.1200.amd64.open x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_tailcallstress
2022-08-07T08:58:31.884Z osx.1200.amd64.open x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress2
2022-08-07T08:43:20.985Z osx.1200.amd64.open x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1

More failures
Queued OS Arch Column1
2022-08-07T08:43:16.875Z osx.1200.amd64.open x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_heapverify1
2022-08-07T08:21:43.589Z osx.1200.amd64.open x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable_jitstress2
2022-08-07T08:21:42.121Z osx.1200.amd64.open x64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_zapdisable
2022-08-07T14:42:29.577Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-07T14:40:21.014Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-06T15:05:23.319Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-06T14:56:37.108Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-07-31T15:21:12.191Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-07-31T15:16:48.028Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-07-30T15:04:39.514Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-07-30T15:03:50.956Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3

v-wenyuxu avatar Aug 08 '22 01:08 v-wenyuxu

I'm moving this to the 8.0.0 milestone. After chatting with @adamsitnik we don't think this is a product issue because we don't have any architecture-specific code in play here. But we'll take a look from the product side after the 7.0.0 RC1 snap.

jeffhandley avatar Aug 10 '22 18:08 jeffhandley

Failed again in: runtime-coreclr gcstress-extra 20220814.1

Failed test:

coreclr OSX arm64 Checked gcstress0xc_zapdisable @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcsvalbox_d/lcsvalbox_d.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable_heapverify1 @ OSX.1200.ARM64.Open

- JIT/Methodical/tailcall_v4/hijacking/hijacking.cmd

Error message:

cmdLine:/private/tmp/helix/working/AD430949/w/AE270949/e/JIT/Methodical/Methodical_d1/../Arrays/lcs/lcsvalbox_d/lcsvalbox_d.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 8/14/2022 4:39:33 PM, end: 8/14/2022 6:09:33 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/AD430949/w/AE270949/uploads/Arrays/lcs/lcsvalbox_d/output.txt
Raw output:
SKIPPING EXECUTION BECAUSE COMPlus_GCStress IS SET

cmdLine:/private/tmp/helix/working/AD430949/w/AE270949/e/JIT/Methodical/Methodical_d1/../Arrays/lcs/lcsvalbox_d/lcsvalbox_d.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 8/14/2022 4:39:33 PM, end: 8/14/2022 6:09:33 PM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/AD430949/w/AE270949/e/JIT/Methodical/Methodical_d1/../Arrays/lcs/lcsvalbox_d/lcsvalbox_d.sh
Expected: True
Actual:   False


Stack trace
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)

v-wenyuxu avatar Aug 16 '22 01:08 v-wenyuxu

Failed again in: runtime-coreclr gcstress0x3-gcstress0xc 20220828.1

Failed test:

coreclr Linux arm Checked gcstress0x3 @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440

- JIT/Regression/JitBlue/Runtime_72845/Runtime_72845/Runtime_72845.sh

coreclr Linux arm64 Checked gcstress0x3 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220824230426-06f234f

- JIT/Regression/JitBlue/Runtime_72845/Runtime_72845/Runtime_72845.sh

coreclr OSX arm64 Checked gcstress0xc @ OSX.1200.ARM64.Open

- JIT/Methodical/Arrays/lcs/lcsvalbox_do/lcsvalbox_do.cmd

Error message:

cmdLine:/root/helix/work/workitem/e/JIT/Regression/JitBlue/Runtime_72845/Runtime_72845/Runtime_72845.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 8/28/2022 3:11:57 PM, end: 8/28/2022 4:11:57 PM)

Return code:      -100
Raw output file:      /root/helix/work/workitem/uploads/Reports/JIT.Regression/JitBlue/Runtime_72845/Runtime_72845/Runtime_72845.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Runtime_72845.dll ''

cmdLine:/root/helix/work/workitem/e/JIT/Regression/JitBlue/Runtime_72845/Runtime_72845/Runtime_72845.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 8/28/2022 3:11:57 PM, end: 8/28/2022 4:11:57 PM)
Test Harness Exitcode is : -100
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/e/JIT/Regression/JitBlue/Runtime_72845/Runtime_72845/Runtime_72845.sh
Expected: True
Actual:   False


Stack trace
   at JIT_Regression._JitBlue_Runtime_72845_Runtime_72845_Runtime_72845_._JitBlue_Runtime_72845_Runtime_72845_Runtime_72845_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

v-wenyuxu avatar Aug 29 '22 01:08 v-wenyuxu

Queued OS Arch Column1
2022-08-28T15:08:37.572Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-28T15:05:04.526Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-27T14:54:10.863Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-27T14:50:41.211Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-21T14:49:30.007Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3

More failures
Queued OS Arch Column1
2022-08-21T14:45:35.793Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-20T14:41:11.016Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-20T14:40:29.533Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-14T14:43:37.663Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-14T14:43:13.121Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-14T14:43:13.121Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-14T14:43:13.121Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-13T15:12:43.919Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-13T15:05:23.979Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-13T15:05:23.979Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-13T15:05:23.979Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-07T14:42:29.577Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-07T14:40:21.014Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-06T15:05:23.319Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-08-06T14:56:37.108Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-07-31T15:21:12.191Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-07-31T15:16:48.028Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-07-30T15:04:39.514Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-07-30T15:03:50.956Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3

v-wenyuxu avatar Aug 29 '22 01:08 v-wenyuxu

Failed again in: runtime-coreclr gcstress-extra 20220904.1 

Failed test:

coreclr OSX arm64 Checked gcstress0xc_zapdisable_heapverify1 @ OSX.1200.ARM64.Open
- JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

coreclr OSX arm64 Checked gcstress0xc_jitstress2 @ OSX.1200.ARM64.Open
- JIT/Methodical/Boxing/boxunbox/KeepAliveBoxOpt/KeepAliveBoxOpt.cmd

coreclr OSX arm64 Checked gcstress0xc_jitstress1 @ OSX.1200.ARM64.Open
- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable_jitstress2 @ OSX.1200.ARM64.Open
- JIT/Methodical/refany/stress1_r/stress1_r.cmd

coreclr windows x86 Checked gcstress0xc_jitstress1 @ Windows.10.Amd64.Open
- baseservices\\TieredCompilation\\BasicTestWithMcj\\BasicTestWithMcj.cmd

coreclr windows x64 Checked gcstress0xc_jitstress1 @ Windows.10.Amd64.Open
- baseservices\\TieredCompilation\\BasicTestWithMcj\\BasicTestWithMcj.cmd

Error message:

cmdLine:/private/tmp/helix/working/B5640950/w/A9200938/e/JIT/Methodical/Methodical_others/../Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 9/4/2022 4:54:52 PM, end: 9/4/2022 6:24:53 PM)

Return code:      -100
Raw output file:      /tmp/helix/working/B5640950/w/A9200938/uploads/Coverage/copy_prop_byref_to_native_int/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/B5640950/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false copy_prop_byref_to_native_int.dll ''
Expected: 100
Actual: 100
END EXECUTION - PASSED

cmdLine:/private/tmp/helix/working/B5640950/w/A9200938/e/JIT/Methodical/Methodical_others/../Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.sh Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 9/4/2022 4:54:52 PM, end: 9/4/2022 6:24:53 PM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/B5640950/w/A9200938/e/JIT/Methodical/Methodical_others/../Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.sh
Expected: True
Actual:   False


Stack trace
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)

v-wenyuxu avatar Sep 07 '22 05:09 v-wenyuxu

Queued OS Arch Column1
2022-08-13T23:14:59.543Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable

v-wenyuxu avatar Sep 07 '22 05:09 v-wenyuxu

Failed again in: runtime-coreclr gcstress-extra 20220917.1

Failed test:

coreclr windows x86 Checked gcstress0xc_zapdisable_heapverify1 @ Windows.10.Amd64.Open
- JIT\\Regression\\CLR-x86-JIT\\V1.2-M01\\b16473\\b16473\\b16473.cmd

coreclr Linux arm64 Checked gcstress0xc_jitstress2 @ (Ubuntu.1804.Arm64.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm64v8-20220824230426-06f234f
- baseservices/exceptions/simple/ParallelCrashMainThread/ParallelCrashMainThread.sh

coreclr windows x86 Checked gcstress0xc_jitminopts_heapverify1 @ Windows.10.Amd64.Open
- JIT\\Regression\\CLR-x86-JIT\\V1.2-M01\\b16473\\b16473\\b16473.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable @ OSX.1200.ARM64.Open
- JIT/Methodical/Arrays/lcs/lcsvalbox_r/lcsvalbox_r.cmd

coreclr OSX arm64 Checked gcstress0xc_tailcallstress @ OSX.1200.ARM64.Open
- JIT/Methodical/Arrays/misc/arrres_il_r/arrres.cmd
- JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable_heapverify1 @ OSX.1200.ARM64.Open
- JIT/Methodical/VT/port/lcs_gcref_d/lcs_gcref_d.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable_jitstress2 @ OSX.1200.ARM64.Open
- JIT/Methodical/Arrays/lcs/lcsvalbox_do/lcsvalbox_do.cmd

coreclr OSX arm64 Checked gcstress0xc_jitstress1 @ OSX.1200.ARM64.Open
- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd

coreclr OSX arm64 Checked gcstress0xc_jitstress2 @ OSX.1200.ARM64.Open
- JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

Error message:

cmdLine:C:\h\w\A8D9094A\w\ABDB09E4\e\JIT\Regression\CLR-x86-JIT\V1.2-M01\b16473\b16473\b16473.cmd Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 9/17/2022 11:36:17 PM, end: 9/18/2022 1:06:17 AM)

Return code:      -100
Raw output file:      C:\h\w\A8D9094A\w\ABDB09E4\uploads\Reports\JIT.Regression\CLR-x86-JIT\V1.2-M01\b16473\b16473\b16473.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\A8D9094A\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  b16473.dll

cmdLine:C:\h\w\A8D9094A\w\ABDB09E4\e\JIT\Regression\CLR-x86-JIT\V1.2-M01\b16473\b16473\b16473.cmd Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 9/17/2022 11:36:17 PM, end: 9/18/2022 1:06:17 AM)
Test Harness Exitcode is : -100
To run the test:

set CORE_ROOT=C:\h\w\A8D9094A\p
C:\h\w\A8D9094A\w\ABDB09E4\e\JIT\Regression\CLR-x86-JIT\V1.2-M01\b16473\b16473\b16473.cmd
Expected: True
Actual:   False


Stack trace
   at JIT_Regression._CLR_x86_JIT_V1_2_M01_b16473_b16473_b16473_._CLR_x86_JIT_V1_2_M01_b16473_b16473_b16473_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

v-wenyuxu avatar Sep 19 '22 02:09 v-wenyuxu

Queued OS Arch Column1
2022-09-18T00:03:12.997Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress2
2022-08-29T08:02:18.821Z osx.1200.arm64.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitstress1
2022-08-22T10:16:31.694Z osx.1200.arm64.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2022-08-08T09:54:22.672Z osx.1200.arm64.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_tailcallstress
2022-08-07T00:28:12.12Z ubuntu.1804.armarch.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_tailcallstress
2022-07-24T09:26:14.44Z osx.1200.arm64.open arm64 runtime-coreclr crossgen2-composite gcstress Checked-gcstress0xc_jitminopts_heapverify1
2022-08-13T23:14:59.543Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable

v-wenyuxu avatar Sep 19 '22 02:09 v-wenyuxu

Failed again in: runtime-coreclr gcstress-extra 20220925.1

Failed test:

coreclr windows x86 Checked gcstress0xc_jitminopts_heapverify1 @ Windows.10.Amd64.Open
- JIT\\Regression\\CLR-x86-JIT\\V1.2-M01\\b16473\\b16473\\b16473.cmd

coreclr windows x86 Checked gcstress0xc_zapdisable_heapverify1 @ Windows.10.Amd64.Open
- JIT\\Regression\\CLR-x86-JIT\\V1.2-M01\\b16473\\b16473\\b16473.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable_jitstress2 @ OSX.1200.ARM64.Open
- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

coreclr OSX arm64 Checked gcstress0xc_jitstress1 @ OSX.1200.ARM64.Open
- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd
- JIT/Methodical/Arrays/lcs/lcs2_r/lcs2_r.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable @ OSX.1200.ARM64.Open
- JIT/Methodical/Arrays/lcs/lcs2_ro/lcs2_ro.cmd

coreclr OSX arm64 Checked gcstress0xc_zapdisable_heapverify1 @ OSX.1200.ARM64.Open
- JIT/Methodical/tailcall_v4/hijacking/hijacking.cmd

Error message:

cmdLine:C:\h\w\C4220A65\w\A4C708DB\e\JIT\Regression\CLR-x86-JIT\V1.2-M01\b16473\b16473\b16473.cmd Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 9/26/2022 1:45:54 AM, end: 9/26/2022 3:15:54 AM)

Return code:      -100
Raw output file:      C:\h\w\C4220A65\w\A4C708DB\uploads\Reports\JIT.Regression\CLR-x86-JIT\V1.2-M01\b16473\b16473\b16473.output.txt
Raw output:
BEGIN EXECUTION
"C:\h\w\C4220A65\p\corerun.exe" -p "System.Reflection.Metadata.MetadataUpdater.IsSupported=false"  b16473.dll

cmdLine:C:\h\w\C4220A65\w\A4C708DB\e\JIT\Regression\CLR-x86-JIT\V1.2-M01\b16473\b16473\b16473.cmd Timed Out (timeout in milliseconds: 5400000 from variable __TestTimeout, start: 9/26/2022 1:45:54 AM, end: 9/26/2022 3:15:54 AM)
Test Harness Exitcode is : -100
To run the test:

set CORE_ROOT=C:\h\w\C4220A65\p
C:\h\w\C4220A65\w\A4C708DB\e\JIT\Regression\CLR-x86-JIT\V1.2-M01\b16473\b16473\b16473.cmd
Expected: True
Actual:   False


Stack trace
   at JIT_Regression._CLR_x86_JIT_V1_2_M01_b16473_b16473_b16473_._CLR_x86_JIT_V1_2_M01_b16473_b16473_b16473_cmd()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Queued OS Arch Column1
2022-08-13T23:14:59.543Z osx.1200.arm64.open arm64 runtime-coreclr gcstress-extra Checked-gcstress0xc_zapdisable

v-wenyuxu avatar Sep 27 '22 02:09 v-wenyuxu

Failed again in: runtime-coreclr gcstress0x3-gcstress0xc 20221016.1

Failed test:

coreclr OSX arm64 Checked gcstress0xc @ OSX.1200.ARM64.Open

- JIT/Methodical/Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.cmd

Error message:

cmdLine:/private/tmp/helix/working/C1540A31/w/9FC608C6/e/JIT/Methodical/Methodical_others/../Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 10/16/2022 8:30:35 AM, end: 10/16/2022 9:30:35 AM)

Return code:      -100
Raw output file:      /tmp/helix/working/C1540A31/w/9FC608C6/uploads/Coverage/copy_prop_byref_to_native_int/output.txt
Raw output:
BEGIN EXECUTION
/tmp/helix/working/C1540A31/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false copy_prop_byref_to_native_int.dll ''
Expected: 100
Actual: 100
END EXECUTION - PASSED

cmdLine:/private/tmp/helix/working/C1540A31/w/9FC608C6/e/JIT/Methodical/Methodical_others/../Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 10/16/2022 8:30:35 AM, end: 10/16/2022 9:30:35 AM)
Test Harness Exitcode is : -100
To run the test:
Set up CORE_ROOT and run.

/private/tmp/helix/working/C1540A31/w/9FC608C6/e/JIT/Methodical/Methodical_others/../Coverage/copy_prop_byref_to_native_int/copy_prop_byref_to_native_int.sh
Expected: True
Actual:   False


Stack trace
   at Xunit.Assert.True(Nullable`1 condition, String userMessage) in /_/src/xunit.assert/Asserts/BooleanAsserts.cs:line 132
   at TestLibrary.OutOfProcessTest.RunOutOfProcessTest(String basePath, String assemblyPath)
   at Program.<Main>$(String[] args)

v-wenyuxu avatar Oct 17 '22 01:10 v-wenyuxu

@jakobbotsch the times that the process hangs OnSigChild is not being called, SetExited() is also not called somewhere else (that is called on OnSigChild->CheckChildren->TryReapChild->SetExited).

This means that _exitedEvent is never Set() and hence the hang on: https://github.com/dotnet/runtime/blob/58a1180ad6421c6f69cf307b57b2496c888585f2/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/ProcessWaitState.Unix.cs#L411

But I also noticed that _isChild is always true for all processes created in CoreclrTestWrapperLib.RunTest(), I'm not sure if that's correct.

So either one of these reasons is true:

  • OnSigChild should never miss execution for a process to successfully exit. Right now is missing on the hang iteration of your repro, for some reason. I suspect is because the native code that is meant to call it fails silently.

  • The time OnSigChild is not being called is by design and _isChild should not be true, code should go through the !_isChild code path and be able to exit.

  • We should be calling SetExited() somewhere else.

@dotnet/area-system-diagnostics-process @tmds @stephentoub @jkotas is OnSigChild a "must execute" method for a process to successfully exit?

jozkee avatar Oct 24 '22 16:10 jozkee

As @jakobbotsch mentioned, the process is in a bad state after execve is called although execve does not return error. https://github.com/dotnet/runtime/blob/cc5ba0994d6e8a6f5e4a63d1c921a68eda4350e8/src/native/libs/System.Native/pal_process.c#L423-L424

Now, that causes that the OnSigChild callback never triggers because native code stops running, for some reason. https://github.com/dotnet/runtime/blob/cc5ba0994d6e8a6f5e4a63d1c921a68eda4350e8/src/libraries/System.Diagnostics.Process/src/System/Diagnostics/Process.Unix.cs#L1090-L1091 I confirmed that by adding File.WriteAllText("/dev/tty", "Some debug message") to OnSigChild and observing that it doesn't print anything to the terminal when the hang occurs.

I also went to pal_signal.c and added printfs to the code that is meant to call OnSigChild and that also doesn't print on hangs https://github.com/dotnet/runtime/blob/cc5ba0994d6e8a6f5e4a63d1c921a68eda4350e8/src/native/libs/System.Native/pal_signal.c#L385-L390

I also attempted to log rather than printing i.e: write to a file from native c, and that confirmed that that code is indeed not executing as nothing is written when the test loop hangs.

I repeated printing and logging for SignalHandlerLoop and SystemNative_RegisterForSigChld, those also don't execute on hangs.

It appears to me that the last thing that executes on native before the hang is SystemNative_ForkAndExecProcess, which is the function calling execve.

This relates with my previous comment as process.WaitForExit(timeout) depends on OnSigChild to be called.

I assume this is some weird interaction between GCStress and execve on macOS arm64. I will stop looking into it and hope that someone more familiar with GCStress can take a look.

cc @janvorli

jozkee avatar Oct 28 '22 20:10 jozkee

Failed again in: runtime-coreclr gcstress0x3-gcstress0xc 20221030.1

Failed test:

coreclr Linux arm Checked gcstress0x3 @ (Ubuntu.1804.Arm32.Open)[email protected]/dotnet-buildtools/prereqs:ubuntu-18.04-helix-arm32v7-bfcd90a-20200121150440

- ilverify/ILVerificationTests.sh

Error message:

cmdLine:/root/helix/work/workitem/e/ilverify/ILVerificationTests.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 10/30/2022 3:23:30 PM, end: 10/30/2022 4:23:30 PM)

Return code:      -100
Raw output file:      /root/helix/work/workitem/uploads/Reports/ilverify/ILVerificationTests.output.txt
Raw output:
BEGIN EXECUTION
/root/helix/work/correlation/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false ILVerificationTests.dll ''

cmdLine:/root/helix/work/workitem/e/ilverify/ILVerificationTests.sh Timed Out (timeout in milliseconds: 3600000 from variable __TestTimeout, start: 10/30/2022 3:23:30 PM, end: 10/30/2022 4:23:30 PM)
Test Harness Exitcode is : -100
To run the test:

set CORE_ROOT=/root/helix/work/correlation
/root/helix/work/workitem/e/ilverify/ILVerificationTests.sh
Expected: True
Actual:   False


Stack trace
   at ilverify._ILVerificationTests_._ILVerificationTests_sh()
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
   at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
Queued OS Arch Column1
2022-10-30T15:19:14.782Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-10-29T15:04:12.955Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-10-26T06:46:07.151Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2_tiered
2022-10-26T06:01:02.055Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2
2022-10-26T06:01:01.03Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1_tiered
2022-10-26T06:00:58.629Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1
More failures
Queued OS Arch Column1
2022-10-01T14:50:43.025Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3
2022-09-24T23:43:46.956Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2022-09-23T06:56:27.552Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-23T06:53:56.591Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-23T06:43:45.893Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-23T06:37:32.811Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress2
2022-09-23T06:35:54.586Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress2
2022-09-23T06:33:57.904Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-23T06:32:48.56Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-23T06:21:02.909Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress1
2022-09-23T06:21:02.018Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-23T06:19:29.873Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1
2022-09-23T06:19:28.895Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-23T06:10:25.719Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-23T06:10:24.781Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress1
2022-09-23T06:10:23.938Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress2
2022-09-23T06:06:53.989Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1
2022-09-23T06:06:53.056Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-23T06:06:50.788Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2
2022-09-23T06:01:53.178Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-23T06:01:52.287Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress2
2022-09-23T06:01:51.583Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress1
2022-09-22T07:26:26.975Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-22T07:09:06.366Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-22T07:04:03.676Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress2
2022-09-22T07:03:41.261Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-22T06:52:24.578Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-22T06:50:53.059Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress2
2022-09-22T06:41:02.753Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-22T06:41:01.815Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress1
2022-09-22T06:38:30.9Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-22T06:37:18.032Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-22T06:37:17.107Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1
2022-09-22T06:23:36.542Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-22T06:23:34.727Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1
2022-09-22T06:23:33.602Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2
2022-09-22T06:19:30.608Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-22T06:19:28.4Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress2
2022-09-22T06:19:26.433Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress1
2022-09-22T06:11:07.948Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress2
2022-09-22T06:11:06.977Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-22T06:11:06.088Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress1
2022-09-21T08:14:22.657Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-21T06:37:07.63Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-21T06:32:36.124Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-21T06:26:06.417Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress2
2022-09-21T06:24:38.412Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress2
2022-09-21T06:15:40.238Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-21T06:15:08.091Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-21T06:12:12.466Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1
2022-09-21T06:12:11.216Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-21T06:09:58.841Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress1
2022-09-21T06:09:57.685Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-21T06:01:51.189Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress1
2022-09-21T06:01:50.043Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress2
2022-09-21T06:01:48.876Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-21T05:51:55.65Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1
2022-09-21T05:51:54.035Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2
2022-09-21T05:51:52.907Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-21T05:51:25.838Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-21T05:51:24.733Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress2
2022-09-21T05:51:23.576Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress1
2022-09-20T08:27:15.611Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-20T07:03:31.507Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-20T06:45:39.659Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-20T06:40:13.923Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress2
2022-09-20T06:38:42.96Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress2
2022-09-20T06:34:29.388Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-20T06:29:15.318Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress2_tiered
2022-09-20T06:25:35.172Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1
2022-09-20T06:25:34.076Z osx.1200.arm64.open arm64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-20T06:22:33.291Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-20T06:22:30.991Z osx.1200.amd64.open x64 runtime-coreclr jitstress Checked-jitstress1
2022-09-20T06:09:56.372Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1
2022-09-20T06:09:55.262Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-20T06:09:52.543Z ubuntu.1804.armarch.open arm runtime-coreclr jitstress Checked-jitstress2
2022-09-20T06:07:43.087Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress2
2022-09-20T06:07:42.071Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-20T06:07:40.552Z ubuntu.1804.amd64.open.rt x64 runtime-coreclr jitstress Checked-jitstress1
2022-09-20T05:58:14.425Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress2
2022-09-20T05:58:12.172Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress1
2022-09-20T05:58:10.781Z ubuntu.1804.armarch.open arm64 runtime-coreclr jitstress Checked-jitstress1_tiered
2022-09-18T23:34:42.245Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2022-09-10T23:43:49.728Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress-extra Checked-gcstress0xc_jitstress1
2022-09-10T15:09:30.962Z ubuntu.1804.armarch.open arm runtime-coreclr gcstress0x3-gcstress0xc Checked-gcstress0x3

v-wenyuxu avatar Oct 31 '22 01:10 v-wenyuxu