runtime icon indicating copy to clipboard operation
runtime copied to clipboard

[libraries-jitstress] System.Numerics.Tensors.Tests fails with AV when `JitStress=2`

Open jakobbotsch opened this issue 1 year ago • 2 comments

{
  "ErrorMessage": "at System.Numerics.Tensors.TensorPrimitives.<InvokeSpanIntoSpan>g__VectorizedSmall",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Example pipeline: https://dev.azure.com/dnceng-public/public/_build/results?buildId=562242&view=results Example console log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-eddc37829cc5439788/System.Numerics.Tensors.Net8.Tests/1/console.6be6415c.log?helixlogtype=result

  Starting:    System.Numerics.Tensors.Tests (parallel test collections = on [2 threads], stop on fail = off)
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Numerics.Tensors.TensorPrimitives.<InvokeSpanIntoSpan>g__VectorizedSmall|229_4[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+MultiplyRoundDivideOperator`2[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+FloorOperator`1[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]](Double ByRef, Double ByRef, UIntPtr, MultiplyRoundDivideOperator`2<Double,FloorOperator`1<Double>>)
   at System.Numerics.Tensors.TensorPrimitives.InvokeSpanIntoSpan[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+MultiplyRoundDivideOperator`2[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+FloorOperator`1[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]](System.ReadOnlySpan`1<Double>, MultiplyRoundDivideOperator`2<Double,FloorOperator`1<Double>>, System.Span`1<Double>)
   at System.Numerics.Tensors.TensorPrimitives.Round[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.ReadOnlySpan`1<Double>, Int32, System.MidpointRounding, System.Span`1<Double>)

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=562242 Error message validated: [at System.Numerics.Tensors.TensorPrimitives.<InvokeSpanIntoSpan>g__VectorizedSmall] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 2/12/2024 10:52:37 AM UTC

Report

Build Definition Test Pull Request
562242 dotnet/runtime System.Numerics.Tensors.Tests.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
1 1 1

jakobbotsch avatar Feb 12 '24 10:02 jakobbotsch

Tagging subscribers to this area: @dotnet/area-system-numerics See info in area-owners.md if you want to be subscribed.

Issue Details
{
  "ErrorMessage": "at System.Numerics.Tensors.TensorPrimitives.<InvokeSpanIntoSpan>g__VectorizedSmall",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Example pipeline: https://dev.azure.com/dnceng-public/public/_build/results?buildId=562242&view=results Example console log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-eddc37829cc5439788/System.Numerics.Tensors.Net8.Tests/1/console.6be6415c.log?helixlogtype=result

  Starting:    System.Numerics.Tensors.Tests (parallel test collections = on [2 threads], stop on fail = off)
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Numerics.Tensors.TensorPrimitives.<InvokeSpanIntoSpan>g__VectorizedSmall|229_4[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+MultiplyRoundDivideOperator`2[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+FloorOperator`1[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]](Double ByRef, Double ByRef, UIntPtr, MultiplyRoundDivideOperator`2<Double,FloorOperator`1<Double>>)
   at System.Numerics.Tensors.TensorPrimitives.InvokeSpanIntoSpan[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+MultiplyRoundDivideOperator`2[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+FloorOperator`1[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]](System.ReadOnlySpan`1<Double>, MultiplyRoundDivideOperator`2<Double,FloorOperator`1<Double>>, System.Span`1<Double>)
   at System.Numerics.Tensors.TensorPrimitives.Round[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.ReadOnlySpan`1<Double>, Int32, System.MidpointRounding, System.Span`1<Double>)
Author: jakobbotsch
Assignees: -
Labels:

area-System.Numerics, blocking-clean-ci-optional, Known Build Error

Milestone: -

ghost avatar Feb 12 '24 10:02 ghost

Tagging subscribers to this area: @JulieLeeMSFT, @jakobbotsch See info in area-owners.md if you want to be subscribed.

Issue Details
{
  "ErrorMessage": "at System.Numerics.Tensors.TensorPrimitives.<InvokeSpanIntoSpan>g__VectorizedSmall",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Example pipeline: https://dev.azure.com/dnceng-public/public/_build/results?buildId=562242&view=results Example console log: https://helixre8s23ayyeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-heads-main-eddc37829cc5439788/System.Numerics.Tensors.Net8.Tests/1/console.6be6415c.log?helixlogtype=result

  Starting:    System.Numerics.Tensors.Tests (parallel test collections = on [2 threads], stop on fail = off)
Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
   at System.Numerics.Tensors.TensorPrimitives.<InvokeSpanIntoSpan>g__VectorizedSmall|229_4[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+MultiplyRoundDivideOperator`2[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+FloorOperator`1[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]](Double ByRef, Double ByRef, UIntPtr, MultiplyRoundDivideOperator`2<Double,FloorOperator`1<Double>>)
   at System.Numerics.Tensors.TensorPrimitives.InvokeSpanIntoSpan[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+MultiplyRoundDivideOperator`2[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[System.Numerics.Tensors.TensorPrimitives+FloorOperator`1[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]], System.Numerics.Tensors, Version=9.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51]](System.ReadOnlySpan`1<Double>, MultiplyRoundDivideOperator`2<Double,FloorOperator`1<Double>>, System.Span`1<Double>)
   at System.Numerics.Tensors.TensorPrimitives.Round[[System.Double, System.Private.CoreLib, Version=9.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]](System.ReadOnlySpan`1<Double>, Int32, System.MidpointRounding, System.Span`1<Double>)
Author: jakobbotsch
Assignees: -
Labels:

area-CodeGen-coreclr, blocking-clean-ci-optional, Known Build Error

Milestone: -

ghost avatar Feb 12 '24 10:02 ghost

cc @tannergooding

BruceForstall avatar Feb 13 '24 17:02 BruceForstall

@EgorBo - were you able to identify? I looked at the dumps and AV seems to be in generated code, but was not able to get all the symbols. In case you are busy, let me know and I can take a look.

kunalspathak avatar Feb 20 '24 19:02 kunalspathak

@EgorBo - were you able to identify? I looked at the dumps and AV seems to be in generated code, but was not able to get all the symbols. In case you are busy, let me know and I can take a look.

Sure, feel free to take over! 🙂

EgorBo avatar Feb 20 '24 20:02 EgorBo

This no longer repros on main. I tried to repro it with artifacts from pipeline and the newly built System.Numerics.Tensor.dll and they don't fail anymore.

kunalspathak avatar Feb 23 '24 18:02 kunalspathak