runtime icon indicating copy to clipboard operation
runtime copied to clipboard

[6.0] Android test failure: Failed to install mobile app

Open carlossanlop opened this issue 2 years ago • 15 comments

Found in the internal->public merge PR for 6.0: https://github.com/dotnet/runtime/pull/83404

Lots of tests affected by the exact same failure message.

  • Queue: Build Android x64 Release AllSubsets_Mono_RuntimeTests
  • Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=204115&view=logs&j=b6f63eff-75cc-59ca-fdbf-d2121601a613&t=92df95dc-ad7c-5fec-a8b4-c8301b7a2225
  • Log file example: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-83404-merge-a043263a5a864fcdaa/JIT/1/console.22d84e79.log?helixlogtype=result
  • Callstack:
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 6.0.14)
  Discovering: JIT.opt.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.opt.XUnitWrapper (found 80 test cases)
  Starting:    JIT.opt.XUnitWrapper (parallel test collections = off, max threads = 2)
    JIT/opt/AssertionPropagation/ArrBoundBinaryOp/ArrBoundBinaryOp.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_opt._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
        
    JIT/opt/AssertionPropagation/ArrBoundMinLength/ArrBoundMinLength.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_opt._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
        
    JIT/opt/AssertionPropagation/ArrBoundUnsigned/ArrBoundUnsigned.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_opt._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
        
    JIT/opt/Casts/shared/shared.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_opt._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

...
and goes on and on...
{
  "ErrorMessage": "The following constructor parameters did not have matching fixture data: _Global globalVar",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}

Report

Build Definition Test Pull Request
533025 dotnet/runtime PayloadGroup0.WorkItemExecution
529936 dotnet/runtime nativeaot.SmokeTests.StackTraceMetadata.WorkItemExecution
529637 dotnet/runtime JIT/Regression/JitBlue/GitHub_1133/GitHub_1133/GitHub_1133.sh
529628 dotnet/runtime baseservices.invalid_operations.WorkItemExecution
529434 dotnet/runtime JIT/HardwareIntrinsics/X86/Avx1/Avx_r/Avx_r.sh
529229 dotnet/runtime GC/API/GC/Collect0/Collect0.sh
529020 dotnet/runtime JIT/opt/AssertionPropagation/ArrBoundBinaryOp/ArrBoundBinaryOp.sh
528325 dotnet/runtime JIT/HardwareIntrinsics/X86/X86Base/CpuId_r/CpuId_r.sh
528297 dotnet/runtime GC/LargeMemory/Regressions/largearraytest/largearraytest.sh
527989 dotnet/runtime baseservices.invalid_operations.WorkItemExecution
526473 dotnet/runtime GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.sh
525794 dotnet/runtime PayloadGroup0.WorkItemExecution
525668 dotnet/runtime GC/Features/HeapExpansion/bestfit-finalize/bestfit-finalize.sh
524795 dotnet/runtime JIT/HardwareIntrinsics/X86/Bmi1.X64/Bmi1.X64_r/Bmi1.X64_r.sh
524794 dotnet/runtime JIT/HardwareIntrinsics/X86/Pclmulqdq/Pclmulqdq_r/Pclmulqdq_r.sh
524760 dotnet/runtime JIT.jit64.opt.WorkItemExecution
524675 dotnet/runtime JIT/Regression/CLR-x86-JIT/V1-M09.5-PDC/b10894/b10894/b10894.sh
524684 dotnet/runtime JIT/HardwareIntrinsics/X86/Popcnt.X64/Popcnt.X64_r/Popcnt.X64_r.sh
524665 dotnet/runtime JIT.Methodical.a-dA-D.WorkItemExecution
524334 dotnet/runtime Loader/classloader/Casting/castclasspointer/castclasspointer.sh
523960 dotnet/runtime nativeaot/SmokeTests/TrimmingBehaviors/TrimmingBehaviors/TrimmingBehaviors.sh dotnet/runtime#96829
522941 dotnet/runtime JIT/HardwareIntrinsics/X86/Aes/Aes_r/Aes_r.sh
522957 dotnet/runtime Loader.classloader.WorkItemExecution
523004 dotnet/runtime PayloadGroup0.WorkItemExecution
522547 dotnet/runtime JIT.jit64.opt.WorkItemExecution dotnet/runtime#96786
522359 dotnet/runtime JIT.Regression.JitBlue.WorkItemExecution
522551 dotnet/runtime JIT/Methodical/eh/basics/emptyfinally_d/emptyfinally_d.sh dotnet/runtime#96783
522205 dotnet/runtime JIT/HardwareIntrinsics/Arm/AdvSimd.Arm64/AdvSimd.Arm64_Part0_r/AdvSimd.Arm64_Part0_r.sh dotnet/runtime#96784
521677 dotnet/runtime baseservices.threading.WorkItemExecution dotnet/runtime#96760
521674 dotnet/runtime baseservices.mono.WorkItemExecution dotnet/runtime#96759
521520 dotnet/runtime JIT/jit64/opt/cg/cgstress/CgStress1_d/CgStress1_d.sh
521104 dotnet/runtime JIT/HardwareIntrinsics/X86/Sse3/LoadAndDuplicateToVector128_r/LoadAndDuplicateToVector128_r.sh
521035 dotnet/runtime PayloadGroup0.WorkItemExecution
521012 dotnet/runtime baseservices.threading.WorkItemExecution
520948 dotnet/runtime JIT/HardwareIntrinsics/X86/Avx2/AddSaturate_r/AddSaturate_r.sh
520798 dotnet/runtime JIT/HardwareIntrinsics/X86/Sse1.X64/Sse.X64_r/Sse.X64_r.sh
520807 dotnet/runtime JIT.Regression.WorkItemExecution dotnet/runtime#96725
520489 dotnet/runtime JIT.HardwareIntrinsics.X86.Sse3.WorkItemExecution
519246 dotnet/runtime Loader.classloader.generics.WorkItemExecution dotnet/runtime#96670
520220 dotnet/runtime nativeaot/SmokeTests/Preinitialization/Preinitialization/Preinitialization.sh
517640 dotnet/runtime nativeaot.SmokeTests.UnitTests.WorkItemExecution
515932 dotnet/runtime nativeaot/SmokeTests/DynamicGenerics/DynamicGenerics/DynamicGenerics.sh
515337 dotnet/runtime nativeaot.SmokeTests.Preinitialization.WorkItemExecution dotnet/runtime#96270
514935 dotnet/runtime JIT/opt/AssertionPropagation/ArrBoundBinaryOp/ArrBoundBinaryOp.sh
512367 dotnet/runtime PayloadGroup0.WorkItemExecution
511815 dotnet/runtime nativeaot.SmokeTests.TrimmingBehaviors.WorkItemExecution
511204 dotnet/runtime PayloadGroup0.WorkItemExecution
510942 dotnet/runtime nativeaot.SmokeTests.StackTraceMetadata.WorkItemExecution
509219 dotnet/runtime JIT/Math/Functions/Functions_r/Functions_r.sh
507905 dotnet/runtime JIT.PGO.WorkItemExecution

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
0 10 50

Known issue validation

Build: :mag_right: Result validation: :warning: Validation could not be done without an Azure DevOps build URL on the issue. Please add it to the "Build: :mag_right:" line. Validation performed at: 6/28/2023 6:30:52 PM UTC

carlossanlop avatar Mar 14 '23 21:03 carlossanlop

Tagging subscribers to 'arch-android': @steveisok, @akoeplinger See info in area-owners.md if you want to be subscribed.

Issue Details

Found in the internal->public merge PR for 6.0: https://github.com/dotnet/runtime/pull/83404

Lots of tests affected by the exact same failure message.

  • Queue: Build Android x64 Release AllSubsets_Mono_RuntimeTests
  • Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=204115&view=logs&j=b6f63eff-75cc-59ca-fdbf-d2121601a613&t=92df95dc-ad7c-5fec-a8b4-c8301b7a2225
  • Log file example: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-83404-merge-a043263a5a864fcdaa/JIT/1/console.22d84e79.log?helixlogtype=result
  • Callstack:
Microsoft.DotNet.XUnitConsoleRunner v2.5.0 (64-bit .NET 6.0.14)
  Discovering: JIT.opt.XUnitWrapper (method display = ClassAndMethod, method display options = None)
  Discovered:  JIT.opt.XUnitWrapper (found 80 test cases)
  Starting:    JIT.opt.XUnitWrapper (parallel test collections = off, max threads = 2)
    JIT/opt/AssertionPropagation/ArrBoundBinaryOp/ArrBoundBinaryOp.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_opt._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
        
    JIT/opt/AssertionPropagation/ArrBoundMinLength/ArrBoundMinLength.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_opt._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
        
    JIT/opt/AssertionPropagation/ArrBoundUnsigned/ArrBoundUnsigned.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_opt._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
        
    JIT/opt/Casts/shared/shared.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_opt._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

...
and goes on and on...
{
  "ErrorMessage": "The following constructor parameters did not have matching fixture data: _Global globalVar",
  "BuildRetry": false,
  "ErrorPattern": "",
  "ExcludeConsoleLog": false
}
Author: carlossanlop
Assignees: -
Labels:

arch-x64, os-android, area-VM-meta-mono

Milestone: -

ghost avatar Mar 14 '23 21:03 ghost

@fanyang-mono does this happen in main as well?

steveisok avatar Mar 15 '23 01:03 steveisok

@steveisok No I haven't seen apps failing to install for quite a while, since we introduced recovery mechanism. I checked that that code doesn't exit in release/6.0 branch.

Comparing https://github.com/dotnet/runtime/blob/release/6.0/src/tests/Common/Coreclr.TestWrapper/MobileAppHandler.cs and https://github.com/dotnet/runtime/blob/main/src/tests/Common/Coreclr.TestWrapper/MobileAppHandler.cs

I am not sure if helix support that feature for 6.0. @premun

fanyang-mono avatar Mar 15 '23 12:03 fanyang-mono

Seems like the version of Arcade on the 6.0 branch does have this backported: https://github.com/dotnet/arcade/blob/0c93c1cb1ef9c9d5c1a59f4ab98c2f7e37f12197/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/xharness-runner.apple.sh

premun avatar Mar 15 '23 13:03 premun

Seems like the version of Arcade on the 6.0 branch does have this backported: https://github.com/dotnet/arcade/blob/0c93c1cb1ef9c9d5c1a59f4ab98c2f7e37f12197/src/Microsoft.DotNet.Helix/Sdk/tools/xharness-runner/xharness-runner.apple.sh

I assume that the functionality for Android is backported as well?

fanyang-mono avatar Mar 16 '23 15:03 fanyang-mono

For Helix SDK, everything would be backported together

premun avatar Mar 16 '23 16:03 premun

Seen again in the latest 6.0 dependency flow PR: https://github.com/dotnet/runtime/pull/83513

  • Queue: Build Android x64 Release AllSubsets_Mono_RuntimeTests
  • Job: https://dev.azure.com/dnceng-public/public/_build/results?buildId=211182&view=logs&j=b6f63eff-75cc-59ca-fdbf-d2121601a613&t=92df95dc-ad7c-5fec-a8b4-c8301b7a2225
  • Log: https://helixre107v0xd1eu3ibi6ka.blob.core.windows.net/dotnet-runtime-refs-pull-83513-merge-e97b6a304a96472ea2/JIT.Methodical.eh/1/console.74b538d3.log?helixlogtype=result
Starting:    JIT.Methodical.XUnitWrapper (parallel test collections = off, max threads = 2)
    JIT/Methodical/eh/basics/emptyfinally_d/emptyfinally_d.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_Methodical._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----
        
    JIT/Methodical/eh/basics/multihandler_d/multihandler_d.sh [FAIL]
      System.AggregateException : One or more errors occurred. (Failed to install mobile app.
      Expected: True
      Actual:   False) (The following constructor parameters did not have matching fixture data: _Global globalVar)
      ---- Failed to install mobile app.
      Expected: True
      Actual:   False
      ---- The following constructor parameters did not have matching fixture data: _Global globalVar
      Stack Trace:
        
        ----- Inner Stack Trace #1 (Xunit.Sdk.TrueException) -----
           at JIT_Methodical._Global..ctor()
        ----- Inner Stack Trace #2 (Xunit.Sdk.TestClassException) -----

carlossanlop avatar Mar 20 '23 19:03 carlossanlop

Still happening: https://github.com/dotnet/runtime/pull/84357

ericstj avatar Apr 06 '23 19:04 ericstj

Still pretty impactful. Affecting most, if not all 6.0 PRs.

carlossanlop avatar Apr 24 '23 19:04 carlossanlop

I will try to backport a fix for this later this week.

fanyang-mono avatar Apr 24 '23 19:04 fanyang-mono

I will try to backport a fix for this later this week.

@fanyang-mono - is this completed ?

SamMonoRT avatar Jul 25 '23 15:07 SamMonoRT

@fanyang-mono - please update with current status and close/move to Future Milestone.

SamMonoRT avatar Aug 11 '23 23:08 SamMonoRT

There isn't one particular PR, but a handful amount that I would need to backport to 6.0 to make these failures go away. There are still a good amount of PR passes Android runs. For example, https://dev.azure.com/dnceng-public/public/_build/results?buildId=370864&view=results.

I've decided to close this issue.

fanyang-mono avatar Aug 14 '23 22:08 fanyang-mono

@fanyang-mono due to the high impact of this failure in servicing branches, I would prefer to keep this open to ensure KnownBuildError links the PR failures with this issue. Saves me tons of investigation time. Example: https://github.com/dotnet/runtime/pull/91078

I moved it to Future instead of 8.0.0.

carlossanlop avatar Aug 24 '23 23:08 carlossanlop

@akoeplinger @lewing @steveisok @radical

Since this was moved to Future, can we please disable the 6.0 CI legs that consistently hit this failure?

carlossanlop avatar Jan 12 '24 03:01 carlossanlop

cc/ @vitek-karas

fanyang-mono avatar Jul 26 '24 16:07 fanyang-mono