marathon icon indicating copy to clipboard operation
marathon copied to clipboard

Published 20 hours ago • verified publisher icon MarathonLabs

Cucumber Android Run on CI failing

Open anas-baadshah opened this issue 2 years ago • 12 comments

Describe the bug Marathon execution failing on CI. Local execution is working fine. When running on CI, It fails with the following

E 12:25:59.962 [main] <com.malinskiy.marathon.Marathon> com.malinskiy.marathon.exceptions.NoTestCasesFoundException: No tests cases were found

Tests are running on Cucumber, We added config for running a specific tag on marathon file like following

  testParserConfiguration:
    type: remote
    instrumentationArgs:
      tags: "@batch-2"
      features: "features"
      glue: "com.app.test"
      monochrome: true

To Reproduce Steps to reproduce the behaviour:

name: "android-cucumber-app-tests"
outputDir: "build/reports/marathon"
isCodeCoverageEnabled: true
screenRecordingPolicy: "ON_ANY"
testClassRegexes:
  - ".*"
retryStrategy:
#  type: "no-retry"
  type: "fixed-quota"
  totalAllowedRetryQuota: 10
  retryPerTestQuota: 2
vendorConfiguration:
  type: "Android"
  applicationApk: "${APK_PATH}"
  testApplicationApk: ${TEST_APK_PATH}
  autoGrantPermission: true
  instrumentationArgs:
    debug: "false"
  applicationPmClear: false
  testApplicationPmClear: false
  vendor: ADAM
  waitForDevicesTimeoutMillis: 1000000
  adbInitTimeoutMillis: 60000
  fileSyncConfiguration:
    pull:
    - relativePath: "Android/data/cucumber.cukeulator"
      aggregationMode: TEST_RUN
  testParserConfiguration:
    type: remote
    instrumentationArgs:
      tags: "@mocked-batch2"
      features: "features"
      glue: "com.app.test"
      monochrome: true
  screenRecordConfiguration:
    preferableRecorderType: "screenshot"
    videoConfiguration:
      enabled: true
      width: 1080
      height: 1920
      bitrateMbps: 2
      timeLimit: 300
    screenshotConfiguration:
      enabled: false
      width: 1080
      height: 1920
      delayMs: 200
deviceInitializationTimeoutMillis: 1000000

Expected behavior Run the tests on emulators created on CI machine (macos-latest)

Logs and reports

▸ D 07:14:20.667 [DeviceMonitor] <AdamDeviceProvider> Device emulator-5554 changed state to CONNECTED
1056
[07:14:20]: ▸ D 07:14:20.668 [DeviceMonitor] <AdamDeviceProvider> Device emulator-5556 changed state to CONNECTED
1057
[07:14:20]: ▸ D 07:14:20.796 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2] <AndroidDevice> Device 127.0.0.1:5037:emulator-5554 booted!
1058
[07:14:21]: ▸ D 07:14:21.043 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5556-1] <AndroidDevice> Device 127.0.0.1:5037:emulator-5556 booted!
1059
[07:14:22]: ▸ D 07:14:22.174 [main] <AndroidAppInstaller> Installing application output to 127.0.0.1:5037:emulator-5554
1060
[07:14:22]: ▸ I 07:14:22.655 [main] <AndroidAppInstaller> Installing com.app, /Users/runner/work/appAndroid/appAndroid/apk/build/app_8.8.1-beta_1616.apk to 127.0.0.1:5037:emulator-5554
1061
[07:14:28]: ▸ D 07:14:28.981 [main] <AndroidDevice> Transferred app_8.8.1-beta_1616.apk to/from 127.0.0.1:5037:emulator-5554. 26921.26 KB/s (170277110 bytes in 6.3250)
1062
[07:14:32]: ▸ D 07:14:32.975 [main] <AndroidDevice> /data/local/tmp/app_8.8.1-beta_1616.apk synced in 3289ms
1063
[07:14:47]: ▸ D 07:14:47.685 [main] <AndroidAppInstaller> Success
1064
[07:14:47]: ▸ D 07:14:47.687 [main] <AndroidAppInstaller> Installing instrumentation package to 127.0.0.1:5037:emulator-5554
1065
[07:14:48]: ▸ I 07:14:48.069 [main] <AndroidAppInstaller> Installing com.app.test, /Users/runner/work/appAndroid/appAndroid/apk/test/app-app-debug-androidTest.apk to 127.0.0.1:5037:emulator-5554
1066
[07:14:49]: ▸ D 07:14:49.030 [main] <AndroidDevice> Transferred app-app-debug-androidTest.apk to/from 127.0.0.1:5037:emulator-5554. 26397.50 KB/s (25368823 bytes in 0.9610)
1067
[07:14:49]: ▸ D 07:14:49.554 [main] <AndroidDevice> /data/local/tmp/app-app-debug-androidTest.apk synced in 449ms
1068
[07:14:52]: ▸ D 07:14:52.951 [main] <AndroidAppInstaller> Success
1069
[07:14:52]: ▸ D 07:14:52.952 [main] <AndroidAppInstaller> Prepare installation finished for 127.0.0.1:5037:emulator-5554
1070
[07:16:10]: ▸ W 07:16:10.872 [main] <c.m.m.android.adam.RemoteTestParser> Bundle /Users/runner/work/appAndroid/appAndroid/apk/test/app-app-debug-androidTest.apk did not report any test annotations. If you need test annotations retrieval, remote test parser requires additional setup see https://marathonlabs.github.io/marathon/ven/android.html#test-parser
1071
[07:16:10]: ▸ E 07:16:10.880 [main] <com.malinskiy.marathon.Marathon> com.malinskiy.marathon.exceptions.NoTestCasesFoundException: No tests cases were found
1072
[07:16:10]: ▸ marathon v0.7.0: Test run failed

Devices (please complete the following information):

  • Device:
 - echo no | avdmanager --verbose create avd -f -n parallel_2 -k "system-images;android-29;google_apis;x86" --tag "google_apis" --abi "x86"
emulator -avd parallel_1 -skin 768x1280 -no-window -gpu swiftshader_indirect -noaudio -no-boot-anim -camera-back none -verbose &
  • OS: Android API 29 emulator

anas-baadshah avatar Apr 07 '22 13:04 anas-baadshah

Hey @anas-baadshah, thanks for submitting this.

Upon a review of your config I want to make sure you understand how the remote parser works for the cucumber

You need to emit the metrics for the names of the tests somehow, an example of this is with the provided adam junit listener by adding to your test apk:

dependecies {
  androidTestImplementation("com.malinskiy.adam:android-junit4-test-annotation-producer:${LATEST_VERSION}")
}

and adding this listener to the execution of marathon's test run:

vendorConfiguration:
  type: "Android"
  testParserConfiguration:
    type: "remote"
    instrumentationArgs:
      listener: "com.malinskiy.adam.junit4.android.listener.TestAnnotationProducer"

Since your snippet doesn't include this listener:

testParserConfiguration:
  type: remote
  instrumentationArgs:
    tags: "@batch-2"
    features: "features"
    glue: "com.app.test"
    monochrome: true

marathon finished the test run because no tests were actually reported:

[07:16:10]: ▸ W 07:16:10.872 [main] <c.m.m.android.adam.RemoteTestParser> Bundle /Users/runner/work/appAndroid/appAndroid/apk/test/app-app-debug-androidTest.apk did not report any test annotations. If you need test annotations retrieval, remote test parser requires additional setup see https://marathonlabs.github.io/marathon/ven/android.html#test-parser
1071

Hope this helps

Malinskiy avatar Apr 11 '22 08:04 Malinskiy

Thank you @Malinskiy, I tried to add this annotation producer to the configuration and it throws an error E 09:46:24.271 [main] <com.malinskiy.marathon.Marathon> java.lang.RuntimeException: Unable to parse test app AndroidManifest.xml.

Btw, Without this listener, Locally it is executing the tests. On CI only it shows the mentioned error. Screen Shot 2022-04-13 at 9 51 10 AM Adding listener is failing on the above step. Do you have any idea why this is happening?

anas-baadshah avatar Apr 13 '22 05:04 anas-baadshah

Maybe you can share the manifest somehow with me and I can try to reproduce it? I've never seen android manifest parsing error tbh.

Alternatively you can just write a test in the marathon's code or create a reproducing androidmanifest without your application-specific details

Malinskiy avatar Apr 13 '22 05:04 Malinskiy

Here is the test android manifest file

<?xml version="1.0" encoding="utf-8"?>
<manifest
    xmlns:android="http://schemas.android.com/apk/res/android"
    android:compileSdkVersion="30"
    android:compileSdkVersionCodename="11"
    package="com.app.test"
    platformBuildVersionCode="30"
    platformBuildVersionName="11">

    <uses-sdk
        android:minSdkVersion="23"
        android:targetSdkVersion="30" />

    <instrumentation
        android:label="Tests for com.app"
        android:name="com.app.test.CucumberInstrumentationRunner"
        android:targetPackage="com.app"
        android:handleProfiling="false"
        android:functionalTest="false" />

    <application
        android:debuggable="true"
        android:extractNativeLibs="false">

        <uses-library
            android:name="android.test.runner" />
    </application>
</manifest>

anas-baadshah avatar Apr 13 '22 08:04 anas-baadshah

When adding the listener on to the marathon file, it fails to install the apks to the emulators and then throwing the error like

[12:54:19]: ▸ D 12:54:18.203 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1] <AndroidDevice> Device 127.0.0.1:5037:emulator-5554 booted!
[12:54:22]: ▸ D 12:54:19.003 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5556-1] <AndroidDevice> Device 127.0.0.1:5037:emulator-5556 booted!
[12:54:22]: ▸ D 12:54:21.420 [main] <c.m.m.android.adam.RemoteTestParser> Remote parsing failed. Retrying
[12:54:22]: ▸ D 12:54:21.421 [main] <c.m.m.android.adam.RemoteTestParser> Unable to parse test app AndroidManifest.xml

[12:55:04]: ▸ E 12:55:04.649 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1] <AndroidDevice>
[12:55:04]: ▸ kotlinx.coroutines.JobCancellationException: Job was cancelled
[12:55:04]: ▸ E 12:55:04.653 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1] <AndroidDevice>
[12:55:04]: ▸ kotlinx.coroutines.JobCancellationException: Job was cancelled
[12:55:04]: ▸ E 12:55:04.654 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1] <AndroidDevice>
[12:55:04]: ▸ kotlinx.coroutines.JobCancellationException: Job was cancelled
[12:55:04]: ▸ E 12:55:04.659 [main] <com.malinskiy.marathon.Marathon> java.lang.RuntimeException: Unable to parse test app AndroidManifest.xml.

anas-baadshah avatar Apr 13 '22 08:04 anas-baadshah

@anas-baadshah I mean the manifest inside the apk, not the source manifest file that is plain text. The android manifest inside the apk goes through some processing and is not human readable.

I've added a sample test on a branch for this: https://github.com/MarathonLabs/marathon/commit/063986325396568748a0cc0a88d5ab8f4f5d9546 I'd like to see the manifest that you have inside the apk file to debug this problem

Malinskiy avatar Apr 14 '22 04:04 Malinskiy

@Malinskiy When i drag and drop the test apk to the AS. it shows this manifest file there. Is there something I am missing? I analyzed the apk in order to find another binary but couldn't find any. Screen Shot 2022-04-14 at 11 07 54 AM

anas-baadshah avatar Apr 14 '22 07:04 anas-baadshah

Just rename the x.apk to x.zip and use any unzip tool. The output should have the manifest in the root

Malinskiy avatar Apr 14 '22 07:04 Malinskiy

Thanks @Malinskiy, Here it is. AndroidManifest.xml.zip

anas-baadshah avatar Apr 14 '22 07:04 anas-baadshah

Reproduced. Looking into the problem

Malinskiy avatar Apr 14 '22 07:04 Malinskiy

The library I'm using to parse the manifest is reading the offset of strings packed in the manifest location 12435439==BDBFEF

when the file has 2k bytes. Something fishy is going on here. Looking at hex comparison to a normal file I see these blobs of BDBFEF happening multiple times. Not sure what these are image

To be fair, androguard also fails to read the same file. Maybe there is a new spec for the manifest? I dunno, will look into this.

 androguard axml ~/Downloads/AndroidManifest.xml/AndroidManifest.xml 
[WARNING ] androguard.axml: Declared filesize (1796) is smaller than total file size (2070). Was something appended to the file? Trying to parse it anyways.
[WARNING ] androguard.axml: Size of strings is not aligned by four bytes.
Traceback (most recent call last):
  File "/home/pkunzip/.local/bin/androguard", line 8, in <module>
    sys.exit(entry_point())
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/lib/python3.10/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/lib/python3.10/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/home/pkunzip/.local/lib/python3.10/site-packages/androguard/cli/entry_points.py", line 91, in axml
    androaxml_main(file_, output, resource)
  File "/home/pkunzip/.local/lib/python3.10/site-packages/androguard/cli/main.py", line 36, in androaxml_main
    axml = AXMLPrinter(read(inp)).get_xml_obj()
  File "/home/pkunzip/.local/lib/python3.10/site-packages/androguard/core/bytecodes/axml/__init__.py", line 928, in __init__
    _type = next(self.axml)
  File "/home/pkunzip/.local/lib/python3.10/site-packages/androguard/core/bytecodes/axml/__init__.py", line 477, in __next__
    self._do_next()
  File "/home/pkunzip/.local/lib/python3.10/site-packages/androguard/core/bytecodes/axml/__init__.py", line 493, in _do_next
    h = ARSCHeader(self.buff)
  File "/home/pkunzip/.local/lib/python3.10/site-packages/androguard/core/bytecodes/axml/__init__.py", line 2074, in __init__
    self._type, self._header_size, self._size = unpack('<HHL', buff.read(self.SIZE))
struct.error: unpack requires a buffer of 8 bytes

The AndroidManifest also can't be read by Android Studio Bumblebee 2021.1.1 P3 image

Malinskiy avatar Apr 14 '22 08:04 Malinskiy

@anas-baadshah Please close this if the issue was resolved since the problem was due to apk corruption

Malinskiy avatar Sep 16 '22 08:09 Malinskiy