dnceng icon indicating copy to clipboard operation
dnceng copied to clipboard

[android] Failing E2E Android - Devices Helix Tests Build_Debug tests

Open ivanpovazan opened this issue 1 year ago • 3 comments

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=750141

Build leg reported

E2E Android - Devices Helix Tests Build_Debug failed

Pull Request

https://github.com/dotnet/xharness/pull/1233

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

{
  "ErrorMessage": "crit: Install failure: Test command cannot continue",
  "BuildRetry": false,
  "ExcludeConsoleLog": false
}

@dotnet/dnceng

Release Note Category

  • [ ] Feature changes/additions
  • [ ] Bug fixes
  • [ ] Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: :mag_right: https://dev.azure.com/dnceng-public/public/_build/results?buildId=750141 Error message validated: [crit: Install failure: Test command cannot continue] Result validation: :white_check_mark: Known issue matched with the provided build. Validation performed at: 7/29/2024 6:26:51 PM UTC

Report

Build Definition Test Pull Request
838677 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1295
838368 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1292
837857 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1291
837837 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1291
836737 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution
836584 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/arcade#15153
833645 dotnet/runtime System.Threading.Tasks.Dataflow.Tests.WorkItemExecution
832523 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/arcade#15146
832047 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1288
832010 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
831971 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1288
825305 dotnet/runtime System.Threading.ThreadPool.Tests.WorkItemExecution
823678 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution
823581 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
822315 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
822026 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1282
822072 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15087
819558 dotnet/runtime System.Security.Cryptography.OpenSsl.Tests.WorkItemExecution
819157 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1279
819143 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
819033 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1277
818848 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1277
818803 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1277
815614 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15087
817968 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1277
815834 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
815712 dotnet/xharness System.Buffers.Tests-x86.WorkItemExecution dotnet/xharness#1276
812840 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
810489 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
810474 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
809758 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1273
809552 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1272
809474 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1271
809392 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/arcade#15087
809353 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1270
809244 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
809163 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1268
808489 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution
808108 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1268
808047 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1267
807953 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1267
807915 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution
807376 dotnet/xharness System.Buffers.Tests-x86.WorkItemExecution
806990 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15020
806807 dotnet/xharness System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/xharness#1266
806688 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution
805484 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15071
805447 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution
805368 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15072
804447 dotnet/arcade System.Buffers.Tests-arm64-v8a.WorkItemExecution dotnet/arcade#15072
804445 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15071
805076 dotnet/arcade System.Buffers.Tests-arm64-v8a-custom-command.WorkItemExecution dotnet/arcade#15063

Summary

24-Hour Hit Count 7-Day Hit Count 1-Month Count
4 11 52

ivanpovazan avatar Jul 29 '24 18:07 ivanpovazan

Problem

This issue seems to be occurring more and more, and xharness CI is mostly red.

The problem comes from the fact that in some jobs we are testing debug apks on physical Android devices, which gives:

adb: failed to install D:\h\w\A8C809B3\w\A92C0917\e\System.Buffers.Tests-arm64-v8a.apk: Failure [INSTALL_FAILED_VERIFICATION_FAILURE: Package Verification Result]
             
[02:24:27] crit: Install failure: Test command cannot continue

(link to the latest failure on DNCENGWIN-007 : https://helixre107v0xdeko0k025g8.blob.core.windows.net/dotnet-xharness-refs-heads-release-90-68d2120ee0c9454f80/System.Buffers.Tests-arm64-v8a.Attempt.3/1/console.03621459.log?helixlogtype=result

Installing such apks can be rejected by the device, if Verify apps over USB setting is enabled on it.

Assumption

My assumption is that the tethered devices were initially configured to have this verification disabled in device settings - allowing debug apks to be installed and tested, and that a factory reset happened recently, which caused switching this setting on again and caused the failures we are now seeing.

Ask

DNCENGWIN-070 had a successful run https://dev.azure.com/dnceng-public/public/_build/results?buildId=811051&view=logs&j=00f02412-5f5e-5b12-0780-d2e57de065ba&t=b1200827-0fdc-54c0-1776-c7b8da453b26 while DNCENGWIN-007 seems to fail constantly. Checking the Verify apps over USB setting on tethered devices on these machines would help us figure out the next steps.

@ilyas1974, could someone check this out?

ivanpovazan avatar Sep 23 '24 09:09 ivanpovazan

Hi Ivan, I have taken DNCENG-007 out of rotation so no additional jobs will use that system while we investigate the failure you mentioned above. I have created https://portal.microsofticm.com/imp/v5/incidents/details/546521338 to have the system in question looked at.

ilyas1974 avatar Sep 23 '24 15:09 ilyas1974

Thank you! Should I add some context on the created incident ticket?

ivanpovazan avatar Sep 23 '24 19:09 ivanpovazan

@ilyas1974 as we fixed https://github.com/dotnet/dnceng/issues/4253 this can be closed now.

ivanpovazan avatar Nov 18 '24 08:11 ivanpovazan