runtime icon indicating copy to clipboard operation
runtime copied to clipboard

Published 20 hours ago • verified publisher icon dotnet

[release/7.0][wasm] Multiple `JIT.*` runtime test failures on rolling builds

Open radical opened this issue 3 years ago • 15 comments

JIT.HardwareIntrinsics.Arm.AdvSimd:

    JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/ABBA093D/uploads/Reports/JIT.HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.output.txt
      Raw output:
      BEGIN EXECUTION
      MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
        AppDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
        TestBinDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r
        ArtifactsBinDir: 
        Generated app bundle at /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
      /datadisks/disk1/work/B2CE09D6/p/dotnet-cli/sdk/7.0.100-rc.1.22425.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj]
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
      > /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._Arm_AdvSimd_AdvSimd_Part6_r_AdvSimd_Part6_r_._Arm_AdvSimd_AdvSimd_Part6_r_AdvSimd_Part6_r_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/B2CE09D6/w/ABBA093D/uploads/Reports/JIT.HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.output.txt
        Raw output:
        BEGIN EXECUTION
        MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
          AppDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
          TestBinDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r
          ArtifactsBinDir: 
          Generated app bundle at /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
        /datadisks/disk1/work/B2CE09D6/p/dotnet-cli/sdk/7.0.100-rc.1.22425.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj]
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
        > /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh

JIT.HardwareIntrinsics.X86.General:

    JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/AF3E0916/uploads/Reports/JIT.HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.output.txt
      Raw output:
      BEGIN EXECUTION
      MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
      /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error : Failed to resolve SDK 'Microsoft.Build.NoTargets/1.0.53'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.
      /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error MSB4236: The SDK 'Microsoft.Build.NoTargets/1.0.53' specified could not be found.
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
      > /datadisks/disk1/work/B2CE09D6/w/AF3E0916/e/JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._X86_General_VectorUnused_r_VectorUnused_r_._X86_General_VectorUnused_r_VectorUnused_r_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/B2CE09D6/w/AF3E0916/uploads/Reports/JIT.HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.output.txt
        Raw output:
        BEGIN EXECUTION
        MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
        /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error : Failed to resolve SDK 'Microsoft.Build.NoTargets/1.0.53'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.
        /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error MSB4236: The SDK 'Microsoft.Build.NoTargets/1.0.53' specified could not be found.
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
        > /datadisks/disk1/work/B2CE09D6/w/AF3E0916/e/JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh
  Finished:    JIT.HardwareIntrinsics.XUnitWrapper

JIT.HardwareIntrinsics.X86.Ssse3:

      Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-aCL7KKV98jx3O3fvRayafjy8_eCkKjpwtyjOVymKcEE'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
         at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
         at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
      
      Return code:      82
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/B25B099A/uploads/Reports/JIT.HardwareIntrinsics/X86/Ssse3/Ssse3_ro/Ssse3_ro.output.txt
...

JIT.Regression.CLR-x86-JIT.V1-M12-M13:

    JIT/Regression/CLR-x86-JIT/V1-M13-RTM/b92289/b92289/b92289.sh [FAIL]
      Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-aCL7KKV98jx3O3fvRayafjy8_eCkKjpwtyjOVymKcEE'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
         at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
         at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)

... followed by lot of failures ...

Report

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 0 0

radical avatar Sep 10 '22 14:09 radical

Tagging subscribers to 'arch-wasm': @lewing See info in area-owners.md if you want to be subscribed.

Issue Details

JIT.HardwareIntrinsics.Arm.AdvSimd:

    JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/ABBA093D/uploads/Reports/JIT.HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.output.txt
      Raw output:
      BEGIN EXECUTION
      MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
        AppDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
        TestBinDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r
        ArtifactsBinDir: 
        Generated app bundle at /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
      /datadisks/disk1/work/B2CE09D6/p/dotnet-cli/sdk/7.0.100-rc.1.22425.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj]
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
      > /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._Arm_AdvSimd_AdvSimd_Part6_r_AdvSimd_Part6_r_._Arm_AdvSimd_AdvSimd_Part6_r_AdvSimd_Part6_r_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/B2CE09D6/w/ABBA093D/uploads/Reports/JIT.HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.output.txt
        Raw output:
        BEGIN EXECUTION
        MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
          AppDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
          TestBinDir: /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r
          ArtifactsBinDir: 
          Generated app bundle at /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/WasmApp/
        /datadisks/disk1/work/B2CE09D6/p/dotnet-cli/sdk/7.0.100-rc.1.22425.9/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj]
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
        > /datadisks/disk1/work/B2CE09D6/w/ABBA093D/e/JIT/HardwareIntrinsics/Arm/AdvSimd/AdvSimd_Part6_r/AdvSimd_Part6_r.sh

JIT.HardwareIntrinsics.X86.General:

    JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/AF3E0916/uploads/Reports/JIT.HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.output.txt
      Raw output:
      BEGIN EXECUTION
      MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
      /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error : Failed to resolve SDK 'Microsoft.Build.NoTargets/1.0.53'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.
      /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error MSB4236: The SDK 'Microsoft.Build.NoTargets/1.0.53' specified could not be found.
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
      > /datadisks/disk1/work/B2CE09D6/w/AF3E0916/e/JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._X86_General_VectorUnused_r_VectorUnused_r_._X86_General_VectorUnused_r_VectorUnused_r_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/B2CE09D6/w/AF3E0916/uploads/Reports/JIT.HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.output.txt
        Raw output:
        BEGIN EXECUTION
        MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
        /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error : Failed to resolve SDK 'Microsoft.Build.NoTargets/1.0.53'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.
        /datadisks/disk1/work/B2CE09D6/p/wasm-test-runner/WasmTestRunner.proj : error MSB4236: The SDK 'Microsoft.Build.NoTargets/1.0.53' specified could not be found.
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/B2CE09D6/p
        > /datadisks/disk1/work/B2CE09D6/w/AF3E0916/e/JIT/HardwareIntrinsics/X86/General/VectorUnused_r/VectorUnused_r.sh
  Finished:    JIT.HardwareIntrinsics.XUnitWrapper

JIT.HardwareIntrinsics.X86.Ssse3:

      Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-aCL7KKV98jx3O3fvRayafjy8_eCkKjpwtyjOVymKcEE'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
         at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
         at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)
      
      Return code:      82
      Raw output file:      /datadisks/disk1/work/B2CE09D6/w/B25B099A/uploads/Reports/JIT.HardwareIntrinsics/X86/Ssse3/Ssse3_ro/Ssse3_ro.output.txt
...

JIT.Regression.CLR-x86-JIT.V1-M12-M13:

    JIT/Regression/CLR-x86-JIT/V1-M13-RTM/b92289/b92289/b92289.sh [FAIL]
      Unhandled exception: System.IO.IOException: Connection timed out : 'Global\msbuild-server-launch-aCL7KKV98jx3O3fvRayafjy8_eCkKjpwtyjOVymKcEE'
         at System.Threading.Mutex.CreateMutexCore(Boolean initiallyOwned, String name, Boolean& createdNew)
         at Microsoft.Build.Experimental.MSBuildClient.TryLaunchServer()
         at Microsoft.Build.Experimental.MSBuildClient.Execute(CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildClientApp.Execute(String[] commandLine, String msbuildLocation, CancellationToken cancellationToken)
         at Microsoft.Build.CommandLine.MSBuildApp.Main(String[] args)
         at Microsoft.DotNet.Cli.Utils.MSBuildForwardingAppWithoutLogging.ExecuteInProc(String[] arguments)

... followed by lot of failures ...
Author: radical
Assignees: -
Labels:

arch-wasm, blocking-clean-ci, area-VM-meta-mono, test-failure
Milestone: -

ghost avatar Sep 10 '22 14:09 ghost

Also on PRs in main, e.g. https://github.com/dotnet/runtime/pull/75318/checks?check_run_id=8279068492 https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75318-merge-3cb5f6cbea0a472187/JIT.HardwareIntrinsics.General.Vector256/1/console.f2f14acf.log?helixlogtype=result

      > /datadisks/disk1/work/B89F09F9/w/BCB50A0C/e/JIT/HardwareIntrinsics/General/Vector256_1/Vector256_1_ro/Vector256_1_ro.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._General_Vector256_1_Vector256_1_ro_Vector256_1_ro_._General_Vector256_1_Vector256_1_ro_Vector256_1_ro_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)

stephentoub avatar Sep 10 '22 15:09 stephentoub

And: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75212-merge-244fe847cfa44a98ac/JIT.HardwareIntrinsics.General.Vector256/1/console.480092cf.log?helixlogtype=result https://dev.azure.com/dnceng-public/public/_build/results?buildId=10368&view=ms.vss-test-web.build-test-results-tab&runId=209152&resultId=102987&paneView=dotnet-dnceng.dnceng-anon-build-release-tasks.helix-anon-test-information-tab

stephentoub avatar Sep 10 '22 15:09 stephentoub

And build, log:

    JIT/Regression/JitBlue/Runtime_74126/Runtime_74126/Runtime_74126.sh [FAIL]
      
      Unhandled Exception:
      System.NullReferenceException: Object reference not set to an instance of an object
         at Runtime_74126.GetVtor[Vector256`1](Vector256`1 vtor)
         at Runtime_74126.Main()
      [ERROR] FATAL UNHANDLED EXCEPTION: System.NullReferenceException: Object reference not set to an instance of an object
         at Runtime_74126.GetVtor[Vector256`1](Vector256`1 vtor)
         at Runtime_74126.Main()
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/B6FB09E0/w/BC8609F2/uploads/Reports/JIT.Regression/JitBlue/Runtime_74126/Runtime_74126/Runtime_74126.output.txt
      Raw output:
      BEGIN EXECUTION
      /datadisks/disk1/work/B6FB09E0/p/corerun -p System.Reflection.Metadata.MetadataUpdater.IsSupported=false Runtime_74126.dll ''
      Expected: 100
      Actual: 1
      END EXECUTION - FAILED
...

radical avatar Sep 10 '22 15:09 radical

~This seems to have come in with the new sdk~

lewing avatar Sep 10 '22 17:09 lewing

cc @rainersigwald

lewing avatar Sep 10 '22 17:09 lewing

https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75220-merge-4f48f0e0781f4dfc9a/JIT.jit64/1/console.8abd5707.log?helixlogtype=result

lewing avatar Sep 10 '22 17:09 lewing

I think we should try setting DOTNET_CLI_DO_NOT_USE_MSBUILD_SERVER=true and see what happens here

lewing avatar Sep 14 '22 20:09 lewing

Are all these JIT failures closely related enough to be grouped into a single issue?

Anyway, I found a bunch of other failures in this release/7.0-rc2 backport PR: https://github.com/dotnet/runtime/pull/75675

  • Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=20221&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552&l=37
  • Queue: Build Browser wasm Linux Release AllSubsets_Mono_RuntimeTests
  • Logs:

Failure log: https://helix.dot.net/api/2019-06-17/jobs/65f2f4ec-4e82-480f-9f8b-ca6855cbff9b/workitems/JIT.1/console

Failure log: https://helix.dot.net/api/2019-06-17/jobs/65f2f4ec-4e82-480f-9f8b-ca6855cbff9b/workitems/JIT.HardwareIntrinsics.Arm.AdvSimd/console

Failure log: https://helix.dot.net/api/2019-06-17/jobs/65f2f4ec-4e82-480f-9f8b-ca6855cbff9b/workitems/JIT.HardwareIntrinsics.Arm.Dp/console

Failure log: https://helix.dot.net/api/2019-06-17/jobs/65f2f4ec-4e82-480f-9f8b-ca6855cbff9b/workitems/JIT.HardwareIntrinsics.General.NotSupported/console

Failure log: https://helix.dot.net/api/2019-06-17/jobs/65f2f4ec-4e82-480f-9f8b-ca6855cbff9b/workitems/JIT.HardwareIntrinsics.X86.Avx1/console

Failure log: https://helix.dot.net/api/2019-06-17/jobs/65f2f4ec-4e82-480f-9f8b-ca6855cbff9b/workitems/JIT.HardwareIntrinsics.X86.Sse2/console

Failure log: https://helix.dot.net/api/2019-06-17/jobs/65f2f4ec-4e82-480f-9f8b-ca6855cbff9b/workitems/JIT.Math/console

Failure log: https://helix.dot.net/api/2019-06-17/jobs/65f2f4ec-4e82-480f-9f8b-ca6855cbff9b/workitems/JIT.Regression/console

carlossanlop avatar Sep 16 '22 22:09 carlossanlop

Also found a bunch of JIT failures in this other release/7.0-rc2 backport PR: https://github.com/dotnet/runtime/pull/75761

  • Queue: Build Browser wasm Linux Release AllSubsets_Mono_RuntimeTests
  • Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=20107&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552&l=37

Logs:

  • https://helix.dot.net/api/2019-06-17/jobs/b84d4776-1b7d-4f72-a1c2-a89dbe0b56d6/workitems/JIT.HardwareIntrinsics.X86.AvxVnni/console
  • https://helix.dot.net/api/2019-06-17/jobs/b84d4776-1b7d-4f72-a1c2-a89dbe0b56d6/workitems/JIT.HardwareIntrinsics.X86.Fma/console
  • https://helix.dot.net/api/2019-06-17/jobs/b84d4776-1b7d-4f72-a1c2-a89dbe0b56d6/workitems/Loader.classloader/console

carlossanlop avatar Sep 16 '22 22:09 carlossanlop

These all appear to be intermittent runtime tests where msbuild exits with a 1 instead of a 0 and there are also often accompanying restore failures on some tests that would have broken in every single test if it was a real issue. The failures also appear to be more related to how we build the wasm runtime tests than actual failures in the tests (if some of these tests were legitimately broken many other tests would have also failed). It is interesting that this seems to be happening much more on the release branches than main now?

lewing avatar Sep 19 '22 02:09 lewing

Happens also for non-JIT tests:

  • 9/16 AM - Rolling run 20115 (7.0-rc2)
    • Loader/classloader/generics/Layout/General/Base01b_seq_ser/Base01b_seq_ser.sh
      • error : Failed to resolve SDK 'Microsoft.Build.NoTargets/3.5.0'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.

karelz avatar Sep 19 '22 15:09 karelz

Seen in this release/7.0-rc2 backport PR: https://github.com/dotnet/runtime/pull/75799

Return code:      1
Raw output file:      /datadisks/disk1/work/B140098C/w/A9580928/uploads/Reports/JIT.HardwareIntrinsics/X86/Pclmulqdq/Pclmulqdq_r/Pclmulqdq_r.output.txt
Raw output:
BEGIN EXECUTION

Welcome to .NET 7.0!
---------------------
SDK Version: 7.0.100-rc.1.22431.11

----------------
Installed an ASP.NET Core HTTPS development certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
Learn about HTTPS: https://aka.ms/dotnet-https
----------------
Write your first app: https://aka.ms/dotnet-hello-world
Find out what's new: https://aka.ms/dotnet-whats-new
Explore documentation: https://aka.ms/dotnet-docs
Report issues and find source on GitHub: https://github.com/dotnet/core
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
--------------------------------------------------------------------------------------
MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
/datadisks/disk1/work/B140098C/p/wasm-test-runner/WasmTestRunner.proj : error : Failed to resolve SDK 'Microsoft.Build.NoTargets/3.5.0'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.
/datadisks/disk1/work/B140098C/p/wasm-test-runner/WasmTestRunner.proj : error MSB4236: The SDK 'Microsoft.Build.NoTargets/3.5.0' specified could not be found.
Test Harness Exitcode is : 1
To run the test:
> set CORE_ROOT=/datadisks/disk1/work/B140098C/p
> /datadisks/disk1/work/B140098C/w/A9580928/e/JIT/HardwareIntrinsics/X86/Pclmulqdq/Pclmulqdq_r/Pclmulqdq_r.sh
Expected: True
Actual:   False

carlossanlop avatar Sep 19 '22 21:09 carlossanlop

I think we should try setting DOTNET_CLI_DO_NOT_USE_MSBUILD_SERVER=true and see what happens here

I agree--has anyone tried this?

also cc @ar-may and @MichalPavlik since this might be MSBuild Server related.

rainersigwald avatar Sep 19 '22 21:09 rainersigwald

Got another hit in a PR updating dependencies for release/7.0-rc2: https://github.com/dotnet/runtime/pull/75675

It's also a JIT HardwareIntrinsics WASM failure, in the Vector128_ro test, but it looks slightly different - it mentions that we are using a preview version of .NET:

message NETSDK1057: You are using a preview version of .NET.

The previous failures I reported were instead complaining that Microsoft.Build.NoTargets wasn't installed. Regardless, in both cases it seems the tests were executed anyway.

Queue: Build Browser wasm Linux Release AllSubsets_Mono_RuntimeTests Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=22359&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552&l=37 Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75675-merge-b703d225d0044e25a7/JIT.HardwareIntrinsics.General.Vector128/1/console.2316287e.log?helixlogtype=result

Callstack 
  Discovering: JIT.HardwareIntrinsics.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.HardwareIntrinsics.XUnitWrapper (found 2 of 362 test cases)
  Starting:    JIT.HardwareIntrinsics.XUnitWrapper (parallel test collections = on, max threads = 2)
    JIT/HardwareIntrinsics/General/Vector128/Vector128_ro/Vector128_ro.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/9E7708C8/w/B80109B2/uploads/Reports/JIT.HardwareIntrinsics/General/Vector128/Vector128_ro/Vector128_ro.output.txt
      Raw output:
      BEGIN EXECUTION
      
      Welcome to .NET 7.0!
      ---------------------
      SDK Version: 7.0.100-rc.1.22431.11
      
      ----------------
      Installed an ASP.NET Core HTTPS development certificate.
      To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
      Learn about HTTPS: https://aka.ms/dotnet-https
      ----------------
      Write your first app: https://aka.ms/dotnet-hello-world
      Find out what's new: https://aka.ms/dotnet-whats-new
      Explore documentation: https://aka.ms/dotnet-docs
      Report issues and find source on GitHub: https://github.com/dotnet/core
      Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
      --------------------------------------------------------------------------------------
      MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
        AppDir: /datadisks/disk1/work/9E7708C8/w/B80109B2/e/JIT/HardwareIntrinsics/General/Vector128/Vector128_ro/WasmApp/
        TestBinDir: /datadisks/disk1/work/9E7708C8/w/B80109B2/e/JIT/HardwareIntrinsics/General/Vector128/Vector128_ro
        ArtifactsBinDir: 
        Generated app bundle at /datadisks/disk1/work/9E7708C8/w/B80109B2/e/JIT/HardwareIntrinsics/General/Vector128/Vector128_ro/WasmApp/
      /datadisks/disk1/work/9E7708C8/p/dotnet-cli/sdk/7.0.100-rc.1.22431.11/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/datadisks/disk1/work/9E7708C8/p/wasm-test-runner/WasmTestRunner.proj]
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
      > /datadisks/disk1/work/9E7708C8/w/B80109B2/e/JIT/HardwareIntrinsics/General/Vector128/Vector128_ro/Vector128_ro.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_HardwareIntrinsics._General_Vector128_Vector128_ro_Vector128_ro_._General_Vector128_Vector128_ro_Vector128_ro_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/9E7708C8/w/B80109B2/uploads/Reports/JIT.HardwareIntrinsics/General/Vector128/Vector128_ro/Vector128_ro.output.txt
        Raw output:
        BEGIN EXECUTION
        
        Welcome to .NET 7.0!
        ---------------------
        SDK Version: 7.0.100-rc.1.22431.11
        
        ----------------
        Installed an ASP.NET Core HTTPS development certificate.
        To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only).
        Learn about HTTPS: https://aka.ms/dotnet-https
        ----------------
        Write your first app: https://aka.ms/dotnet-hello-world
        Find out what's new: https://aka.ms/dotnet-whats-new
        Explore documentation: https://aka.ms/dotnet-docs
        Report issues and find source on GitHub: https://github.com/dotnet/core
        Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli
        --------------------------------------------------------------------------------------
        MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
          AppDir: /datadisks/disk1/work/9E7708C8/w/B80109B2/e/JIT/HardwareIntrinsics/General/Vector128/Vector128_ro/WasmApp/
          TestBinDir: /datadisks/disk1/work/9E7708C8/w/B80109B2/e/JIT/HardwareIntrinsics/General/Vector128/Vector128_ro
          ArtifactsBinDir: 
          Generated app bundle at /datadisks/disk1/work/9E7708C8/w/B80109B2/e/JIT/HardwareIntrinsics/General/Vector128/Vector128_ro/WasmApp/
        /datadisks/disk1/work/9E7708C8/p/dotnet-cli/sdk/7.0.100-rc.1.22431.11/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.RuntimeIdentifierInference.targets(219,5): message NETSDK1057: You are using a preview version of .NET. See: https://aka.ms/dotnet-support-policy [/datadisks/disk1/work/9E7708C8/p/wasm-test-runner/WasmTestRunner.proj]
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/9E7708C8/p
        > /datadisks/disk1/work/9E7708C8/w/B80109B2/e/JIT/HardwareIntrinsics/General/Vector128/Vector128_ro/Vector128_ro.sh
  Finished:    JIT.HardwareIntrinsics.XUnitWrapper
=== TEST EXECUTION SUMMARY ===
   JIT.HardwareIntrinsics.XUnitWrapper  Total: 2, Errors: 0, Failed: 1, Skipped: 0, Time: 9.627s

carlossanlop avatar Sep 19 '22 22:09 carlossanlop

More hits:

  • release/7.0-rc2 backport PR: https://github.com/dotnet/runtime/pull/75875

  • Queue: Build Browser wasm Linux Release AllSubsets_Mono_RuntimeTests

  • Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=23126&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552&l=37

  • Failure logs:

    • https://helix.dot.net/api/2019-06-17/jobs/312c181a-d25b-4299-80f7-8a07d36cd0b7/workitems/JIT.1/console
    • https://helix.dot.net/api/2019-06-17/jobs/312c181a-d25b-4299-80f7-8a07d36cd0b7/workitems/JIT.2/console
    • https://helix.dot.net/api/2019-06-17/jobs/312c181a-d25b-4299-80f7-8a07d36cd0b7/workitems/JIT.HardwareIntrinsics.Arm.ArmBase/console
    • https://helix.dot.net/api/2019-06-17/jobs/312c181a-d25b-4299-80f7-8a07d36cd0b7/workitems/JIT.HardwareIntrinsics.General.Vector128/console
    • https://helix.dot.net/api/2019-06-17/jobs/312c181a-d25b-4299-80f7-8a07d36cd0b7/workitems/JIT.HardwareIntrinsics.X86.Aes/console

carlossanlop avatar Sep 20 '22 18:09 carlossanlop

Happens also for non-JIT tests:

  • 9/16 AM - Rolling run 20115 (7.0-rc2)

    • Loader/classloader/generics/Layout/General/Base01b_seq_ser/Base01b_seq_ser.sh

      • error : Failed to resolve SDK 'Microsoft.Build.NoTargets/3.5.0'. Package restore was successful but a package with the ID of "Microsoft.Build.NoTargets" was not installed.

yeah, it is always tests using the runtime test build stuff which spawns a bunch of builds with slightly different options

lewing avatar Sep 20 '22 19:09 lewing

@lewing did ya'll end up trying disabling msbuild server? If server is potentially the issue here, it'd be good to find that out, get Michal's change into msbuild, get that to flow to an sdk build, and try out that new SDK build all asap as msbuild server is a very big change and if it's causing any reliability problems, we want to sort those out.

marcpopMSFT avatar Sep 21 '22 18:09 marcpopMSFT

Here's another from a release/7.0 PR: https://github.com/dotnet/runtime/pull/75985

  • Queue: Build Browser wasm Linux Release AllSubsets_Mono_RuntimeTests
  • Job results: https://dev.azure.com/dnceng-public/public/_build/results?buildId=25296&view=logs&j=190ad6c8-5950-568c-cadd-f2dfb7d5a79f&t=c0f6fdc1-ac5d-583c-8ae1-a18de0846552&l=37
  • Log file: https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-75985-merge-8223a470ef1f42abab/JIT.1/1/console.e192d364.log?helixlogtype=result
  • Callstack:
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 7.0.0-rc.1.22423.16)
  Discovering: JIT.RyuJIT.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.RyuJIT.XUnitWrapper (found 1 test case)
  Starting:    JIT.RyuJIT.XUnitWrapper (parallel test collections = on, max threads = 2)
  Finished:    JIT.RyuJIT.XUnitWrapper
  Discovering: JIT.superpmi.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.superpmi.XUnitWrapper (found 0 test cases)
  Discovering: JIT.opt.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.opt.XUnitWrapper (found 123 test cases)
  Starting:    JIT.opt.XUnitWrapper (parallel test collections = on, max threads = 2)
    JIT/opt/Types/Equality/Equality.sh [FAIL]
      
      Return code:      1
      Raw output file:      /datadisks/disk1/work/A6DD08DE/w/A4150976/uploads/Reports/JIT.opt/Types/Equality/Equality.output.txt
      Raw output:
      BEGIN EXECUTION
      MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
        AppDir: /datadisks/disk1/work/A6DD08DE/w/A4150976/e/JIT/opt/Types/Equality/WasmApp/
        TestBinDir: /datadisks/disk1/work/A6DD08DE/w/A4150976/e/JIT/opt/Types/Equality
        ArtifactsBinDir: 
      Test Harness Exitcode is : 1
      To run the test:
      > set CORE_ROOT=/datadisks/disk1/work/A6DD08DE/p
      > /datadisks/disk1/work/A6DD08DE/w/A4150976/e/JIT/opt/Types/Equality/Equality.sh
      Expected: True
      Actual:   False
      Stack Trace:
           at JIT_opt._Types_Equality_Equality_._Types_Equality_Equality_sh()
           at System.RuntimeMethodHandle.InvokeMethod(Object target, Void** arguments, Signature sig, Boolean isConstructor)
           at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr)
      Output:
        
        Return code:      1
        Raw output file:      /datadisks/disk1/work/A6DD08DE/w/A4150976/uploads/Reports/JIT.opt/Types/Equality/Equality.output.txt
        Raw output:
        BEGIN EXECUTION
        MSBuild version 17.4.0-preview-22416-02+5d102ae37 for .NET
          AppDir: /datadisks/disk1/work/A6DD08DE/w/A4150976/e/JIT/opt/Types/Equality/WasmApp/
          TestBinDir: /datadisks/disk1/work/A6DD08DE/w/A4150976/e/JIT/opt/Types/Equality
          ArtifactsBinDir: 
        Test Harness Exitcode is : 1
        To run the test:
        > set CORE_ROOT=/datadisks/disk1/work/A6DD08DE/p
        > /datadisks/disk1/work/A6DD08DE/w/A4150976/e/JIT/opt/Types/Equality/Equality.sh
  Finished:    JIT.opt.XUnitWrapper
  Discovering: JIT.CheckProjects.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.CheckProjects.XUnitWrapper (found 0 test cases)
  Discovering: JIT.Performance.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.Performance.XUnitWrapper (found 82 test cases)
  Starting:    JIT.Performance.XUnitWrapper (parallel test collections = on, max threads = 2)
  Finished:    JIT.Performance.XUnitWrapper
=== TEST EXECUTION SUMMARY ===
   JIT.CheckProjects.XUnitWrapper.dll  Total:   0
   JIT.opt.XUnitWrapper                Total: 123, Errors: 0, Failed: 1, Skipped: 0, Time:  116.651s
   JIT.Performance.XUnitWrapper        Total:  82, Errors: 0, Failed: 0, Skipped: 0, Time: 1046.501s
   JIT.RyuJIT.XUnitWrapper             Total:   1, Errors: 0, Failed: 0, Skipped: 0, Time:    3.573s
   JIT.superpmi.XUnitWrapper.dll       Total:   0
                                              ---          -          -           -        ---------
                                 GRAND TOTAL: 206          0          1           0        1166.726s (1166.917s)

carlossanlop avatar Sep 21 '22 22:09 carlossanlop

@lewing did ya'll end up trying disabling msbuild server? If server is potentially the issue here, it'd be good to find that out, get Michal's change into msbuild, get that to flow to an sdk build, and try out that new SDK build all asap as msbuild server is a very big change and if it's causing any reliability problems, we want to sort those out.

@radical can you check the timeline for when we did this and any recent failures?

lewing avatar Sep 23 '22 18:09 lewing

There is rather lots of noise in the replies ... hard to follow. @carlossanlop we will need to figure out better way to track occurrences. It is super-hard to figure out which builds are reported, how often it happened, etc.

@vargaz key question: Is this issue primarily tracking the problem with msbuild returning error code 1?

Test Harness Exitcode is : 1

How far are we from resolution? This seems to be hitting CI quite a bit :( ... causing lots of noise, making it hard to look at other results. If there is not a quick fix, we should consider disabling wasm runs entirely as nobody is going to go deeper into the real issues anyway at this point :(

karelz avatar Sep 23 '22 18:09 karelz

@carlossanlop we will need to figure out better way to track occurrences. It is super-hard to figure out which builds are reported, how often it happened, etc.

That's what I said originally :( too many issues grouped into a single one. If I have time, I'll split them into separate ones later.

carlossanlop avatar Sep 23 '22 18:09 carlossanlop

have we seen this in main after https://github.com/dotnet/runtime/pull/75743 ?

lewing avatar Sep 23 '22 18:09 lewing

I just started updating top post with occurrences from Test Monitoring. I never saw this particular category in main. First 7.0 occurrences happened on 9/15.

karelz avatar Sep 23 '22 18:09 karelz

There have been multiple roslyn/msbuild issues (and some caused by ubuntu's openSSL). I say we take https://github.com/dotnet/runtime/pull/76090 for now and see if things go green. The failures here are definitely build related, if op_Equality was actually failing intermittently we'd see it in all the tests not just these.

lewing avatar Sep 23 '22 18:09 lewing

@jkoritzinsky @trylek can you help walk @rokonec through how the runtime tests build for wasm and why they might trigger interesting failures in the build/compiler servers

lewing avatar Sep 23 '22 19:09 lewing

if these are all on ubuntu 1804 it may be the openssl issue https://github.com/dotnet/runtime/issues/48411#issuecomment-1178405101 and https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1983100 but I think that had a specific exit code

lewing avatar Sep 23 '22 19:09 lewing

@rokonec I can also help walkthrough how the wasm runtime tests are built.

I think some of these errors may have been fixed by https://github.com/dotnet/runtime/pull/75939.

Honestly reading through the issue I am a little confused about what failure we are actually trying to resolve?

naricc avatar Sep 23 '22 19:09 naricc

@rokonec A rough description of the wasm runtime tests. Each test is an individual project. When the tests are built, a xunit wrapper and bash wrapper are generated. When run, the xunit wrapper calls the shell script, and the shell script calls msbuild on this project to actually bundle all the assemblies into a wasm app. Finally, it runs the app. Note the oddity that we are doing a build step at test run time, so this all happens on helix.

The actual command to build the wasm app is generated from this part of the template: https://github.com/dotnet/runtime/blob/d0559e6e33906e4e6db675e646b47ab36eb5bffc/src/tests/Common/CLRTest.Execute.Bash.targets#L292

Are there other details I can provide?

naricc avatar Sep 23 '22 19:09 naricc

I'll add that often the projects are the same code with slightly different msbuild properties

lewing avatar Sep 23 '22 19:09 lewing