mobile: find better way of running regress tests

[bassosimone]

I don't like the current situation where the travis build takes ages plus it fails. Perhaps a good starting point could be to only build some architecture. But I really feel like we're following a wrong path here and I'd rather check what others do (e.g. what do signal do for ios?) and copy them. Because when it is too hard to do something, it's not totally unlikely that we're following weak starting assumptions.

[lorenzoPrimi]

@bassosimone what about this^

[bassosimone]

I have not time to research into this. But I strongly feel that we should have a working CI infrastructure for the apps, because in my experience it helps a lot.

[lorenzoPrimi]

@bassosimone is this still relevant?

[hellais]

@lorenzoPrimi I think this is still relevant and related to: https://github.com/TheTorProject/ooniprobe-ios/issues/121

[hellais]

I have been working on quite some improvements to the iOS testing system via end to end tests in: https://github.com/ooni/probe-ios/pull/195.

The tests, though, is only part of this, I think. The other bit is ensuring we start following some best practices, namely that we should never land anything into master until the end to end tests are green.

[bassosimone]

@hellais Thank you!

[lorenzoPrimi]

@hellais if this is related to appium so should be priority highest

[hellais]

I made a bit of progress on testing Android here: https://github.com/ooni/probe-android/pull/267.

The conclusion I arrived at is that it's probably better to have the testing environment as close as possible to what I have on my laptop, so I can debug it easier. As such I went for the macOS base image and did several tests there.

Here is what I learned:

1. Initially I was using the x86 emulator, because I figured that was going to work better on the VM, but I was wrong. That does not work because it requires hardware acceleration which is not supported. See: https://travis-ci.com/ooni/probe-android/builds/143361097#L378

2. You need a fairly old version of java in order to work with the android sdk. Using recent versions of java results in erratic failure conditions and odd bugs. See: https://travis-ci.com/ooni/probe-android/builds/142868447#L1256. The errors Exception in thread "main" java.lang.NoClassDefFoundError: javax/xml/bind/annotation/XmlSchema looked initially quite criptic.

3. To further on 2. the version matrix of particular android sdk version + OS version + java version seems to have only very small set of compatible options.

After a bunch of pain, I am now running the tests, but I get this error:

Suite for running tests > Android Regression Tests STANDARD_OUT
    Starting Appium server ...
probe.mobile.appium.runners.LandingPageRunner.prepareApp FAILED
    io.appium.java_client.service.local.InvalidServerInstanceException
        Caused by: java.io.IOException
probe.mobile.appium.runners.LandingPageRunner.stopApp FAILED
    java.lang.NullPointerException
3 tests completed, 2 failed, 1 skipped
> Task :androidRegression FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':androidRegression'.
> There were failing tests. See the report at: file:///Users/travis/build/ooni/probe-android/probe-mobile-appium/build/reports/tests/androidRegression/index.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
Deprecated Gradle features were used in this build, making it incompatible with Gradle 7.0.
Use '--warning-mode all' to show the individual deprecation warnings.
See https://docs.gradle.org/6.0.1/userguide/command_line_interface.html#sec:command_line_warnings
BUILD FAILED in 53s
3 actionable tasks: 3 executed
The command "scripts/travis_appium_test.sh" exited with 1.

Which I believe it basically the same error we were getting before, but I think that at least I have understood a bit better the problem.

Probably some more sessions of debugging are needed in order to complete this task.

I should also point out that I this task has not been my main priority during this sprint, but just something I have been doing in the background.

[hellais]

I made no progress on this during the last sprint. Bumping to the next-next one.

[hellais]

@aanorbel what do you think are the next steps for this? Is this issue still relevant?

I am going to un-assign myself and bump down the priority.