android
android copied to clipboard
eduvpn
Please can you document how to use the automated tests?
This project and some sub-projects have automated tests. Please could you document which tests you recommend running and when, and how to check the test results for these sets of tests. I'd like to look deeper than Gradle simply saying BUILD SUCCESSFUL as it's not clear whether it actually ran the tests, nor how many were run, etc.
Thank you.
Hi @julianharty,
The tests are not documented because they are regular Android instrumented unit tests. You can read more about them here: https://developer.android.com/training/testing/unit-testing/instrumented-unit-tests
If you are running the project with Android Studio, you can run the tests by unfolding the file app/src/androidTest/java/nl/eduvpn/app/UnitTestSuite.java. Then click with the right button on that file, and select Run 'UnitTestSuite'.
If you are using the command line, you can run them by executing the command ./gradlew :app:connectedAndroidTest.
See this documentation for more info: https://developer.android.com/studio/test/command-line
If you are using the command line, you can run them by executing the command
./gradlew :app:connectedAndroidTest. See this documentation for more info: https://developer.android.com/studio/test/command-line
$ ./gradlew :app:connectedAndroidTest
Starting a Gradle Daemon (subsequent builds will be faster)
FAILURE: Build failed with an exception.
* What went wrong:
Task 'connectedAndroidTest' not found in project ':app'.
* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 13s
Doesn't seem there is any task similar to what you write above. So how to run the tests? I want to add it to the build_app.sh script as well...
No.
$ ./gradlew app:connectedBasicDebugAndroidTest
FAILURE: Build failed with an exception.
* What went wrong:
Task 'connectedBasicDebugAndroidTest' not found in project ':app'.
* Try:
Run gradlew tasks to get a list of available tasks. Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 1s
$ ./gradlew app:tasks
> Task :app:tasks
------------------------------------------------------------
Tasks runnable from project :app
------------------------------------------------------------
Android tasks
-------------
sourceSets - Prints out all the source sets defined in this project.
Build tasks
-----------
assemble - Assembles all variants of all applications and secondary packages.
assembleAndroidTest - Assembles all the Test applications.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
clean - Deletes the build directory.
cleanBuildCache - Deletes the build cache directory.
Help tasks
----------
buildEnvironment - Displays all buildscript dependencies declared in project ':app'.
components - Displays the components produced by project ':app'. [incubating]
dependencies - Displays all dependencies declared in project ':app'.
dependencyInsight - Displays the insight into a specific dependency in project ':app'.
dependentComponents - Displays the dependent components of components in project ':app'. [incubating]
help - Displays a help message.
model - Displays the configuration model of project ':app'. [incubating]
projects - Displays the sub-projects of project ':app'.
properties - Displays the properties of project ':app'.
tasks - Displays the tasks runnable from project ':app'.
Install tasks
-------------
uninstallAll - Uninstall all applications.
Verification tasks
------------------
check - Runs all checks.
connectedCheck - Runs all device checks on currently connected devices.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
Rules
-----
Pattern: clean<TaskName>: Cleans the output files of a task.
Pattern: build<ConfigurationName>: Assembles the artifacts of a configuration.
Pattern: upload<ConfigurationName>: Assembles and uploads the artifacts belonging to a configuration.
To see all tasks and more detail, run gradlew tasks --all
To see more detail about a task, run gradlew help --task <task>
BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed
There's a ./gradlew in ics-openvpn/ too, that's why I'm asking :)
And if you just run:
./gradlew tasks
$ ./gradlew tasks
> Task :tasks
------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------
Android tasks
-------------
sourceSets - Prints out all the source sets defined in this project.
Build tasks
-----------
assemble - Assembles all variants of all applications and secondary packages.
assembleAndroidTest - Assembles all the Test applications.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
clean - Deletes the build directory.
cleanBuildCache - Deletes the build cache directory.
Build Setup tasks
-----------------
init - Initializes a new Gradle build.
wrapper - Generates Gradle wrapper files.
Help tasks
----------
buildEnvironment - Displays all buildscript dependencies declared in root project 'eduvpn-android-20191126105955'.
components - Displays the components produced by root project 'eduvpn-android-20191126105955'. [incubating]
dependencies - Displays all dependencies declared in root project 'eduvpn-android-20191126105955'.
dependencyInsight - Displays the insight into a specific dependency in root project 'eduvpn-android-20191126105955'.
dependentComponents - Displays the dependent components of components in root project 'eduvpn-android-20191126105955'. [incubating]
help - Displays a help message.
kotlinDslAccessorsReport - Prints the Kotlin code for accessing the currently available project extensions and conventions.
model - Displays the configuration model of root project 'eduvpn-android-20191126105955'. [incubating]
projects - Displays the sub-projects of root project 'eduvpn-android-20191126105955'.
properties - Displays the properties of root project 'eduvpn-android-20191126105955'.
tasks - Displays the tasks runnable from root project 'eduvpn-android-20191126105955' (some of the displayed tasks may belong to subprojects).
Install tasks
-------------
uninstallAll - Uninstall all applications.
Verification tasks
------------------
check - Runs all checks.
connectedCheck - Runs all device checks on currently connected devices.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
To see all tasks and more detail, run gradlew tasks --all
To see more detail about a task, run gradlew help --task <task>
BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed
Yes, I get totally different outputs:
Daniels-MacBook-Pro-2:android dzolnai$ ./gradlew tasks
> Task :tasks
------------------------------------------------------------
Tasks runnable from root project
------------------------------------------------------------
Android tasks
-------------
androidDependencies - Displays the Android dependencies of the project.
signingReport - Displays the signing info for the base and test modules
sourceSets - Prints out all the source sets defined in this project.
Build tasks
-----------
assemble - Assemble main outputs for all the variants.
assembleAndroidTest - Assembles all the Test applications.
assembleBasic - Assembles main outputs for all Basic variants.
assembleDebug - Assembles main outputs for all Debug variants.
assembleDev - Assembles main outputs for all Dev variants.
assembleHome - Assembles main outputs for all Home variants.
assembleRelease - Assembles main outputs for all Release variants.
assembleSkeleton - Assembles main outputs for all Skeleton variants.
assembleUi - Assembles main outputs for all Ui variants.
build - Assembles and tests this project.
buildDependents - Assembles and tests this project and all projects that depend on it.
buildNeeded - Assembles and tests this project and all projects it depends on.
bundle - Assemble bundles for all the variants.
bundleBasic - Assembles bundles for all Basic variants.
bundleDebug - Assembles bundles for all Debug variants.
bundleDev - Assembles bundles for all Dev variants.
bundleHome - Assembles bundles for all Home variants.
bundleRelease - Assembles bundles for all Release variants.
clean - Deletes the build directory.
cleanBuildCache - Deletes the build cache directory.
compileBasicDebugAndroidTestSources
compileBasicDebugSources
compileBasicDebugUnitTestSources
compileBasicReleaseSources
compileBasicReleaseUnitTestSources
compileDevDebugAndroidTestSources
compileDevDebugSources
compileDevDebugUnitTestSources
compileDevReleaseSources
compileDevReleaseUnitTestSources
compileHomeDebugAndroidTestSources
compileHomeDebugSources
compileHomeDebugUnitTestSources
compileHomeReleaseSources
compileHomeReleaseUnitTestSources
compileSkeletonDebugAndroidTestSources
compileSkeletonDebugSources
compileSkeletonDebugUnitTestSources
compileSkeletonReleaseSources
compileSkeletonReleaseUnitTestSources
compileUiDebugAndroidTestSources
compileUiDebugSources
compileUiDebugUnitTestSources
compileUiReleaseSources
compileUiReleaseUnitTestSources
extractSkeletonDebugAnnotations - Extracts Android annotations for the skeletonDebug variant into the archive file
extractSkeletonReleaseAnnotations - Extracts Android annotations for the skeletonRelease variant into the archive file
extractUiDebugAnnotations - Extracts Android annotations for the uiDebug variant into the archive file
extractUiReleaseAnnotations - Extracts Android annotations for the uiRelease variant into the archive file
Build Setup tasks
-----------------
init - Initializes a new Gradle build.
wrapper - Generates Gradle wrapper files.
Cleanup tasks
-------------
lintFix - Runs lint on all variants and applies any safe suggestions to the source code.
Help tasks
----------
buildEnvironment - Displays all buildscript dependencies declared in root project 'android'.
components - Displays the components produced by root project 'android'. [incubating]
dependencies - Displays all dependencies declared in root project 'android'.
dependencyInsight - Displays the insight into a specific dependency in root project 'android'.
dependentComponents - Displays the dependent components of components in root project 'android'. [incubating]
help - Displays a help message.
kotlinDslAccessorsReport - Prints the Kotlin code for accessing the currently available project extensions and conventions.
model - Displays the configuration model of root project 'android'. [incubating]
projects - Displays the sub-projects of root project 'android'.
properties - Displays the properties of root project 'android'.
tasks - Displays the tasks runnable from root project 'android' (some of the displayed tasks may belong to subprojects).
Install tasks
-------------
installBasicDebug - Installs the DebugBasic build.
installBasicDebugAndroidTest - Installs the android (on device) tests for the BasicDebug build.
installDevDebug - Installs the DebugDev build.
installDevDebugAndroidTest - Installs the android (on device) tests for the DevDebug build.
installHomeDebug - Installs the DebugHome build.
installHomeDebugAndroidTest - Installs the android (on device) tests for the HomeDebug build.
installSkeletonDebugAndroidTest - Installs the android (on device) tests for the SkeletonDebug build.
installUiDebugAndroidTest - Installs the android (on device) tests for the UiDebug build.
uninstallAll - Uninstall all applications.
uninstallBasicDebug - Uninstalls the DebugBasic build.
uninstallBasicDebugAndroidTest - Uninstalls the android (on device) tests for the BasicDebug build.
uninstallBasicRelease - Uninstalls the ReleaseBasic build.
uninstallDevDebug - Uninstalls the DebugDev build.
uninstallDevDebugAndroidTest - Uninstalls the android (on device) tests for the DevDebug build.
uninstallDevRelease - Uninstalls the ReleaseDev build.
uninstallHomeDebug - Uninstalls the DebugHome build.
uninstallHomeDebugAndroidTest - Uninstalls the android (on device) tests for the HomeDebug build.
uninstallHomeRelease - Uninstalls the ReleaseHome build.
uninstallSkeletonDebugAndroidTest - Uninstalls the android (on device) tests for the SkeletonDebug build.
uninstallUiDebugAndroidTest - Uninstalls the android (on device) tests for the UiDebug build.
Verification tasks
------------------
check - Runs all checks.
connectedAndroidTest - Installs and runs instrumentation tests for all flavors on connected devices.
connectedBasicDebugAndroidTest - Installs and runs the tests for basicDebug on connected devices.
connectedCheck - Runs all device checks on currently connected devices.
connectedDevDebugAndroidTest - Installs and runs the tests for devDebug on connected devices.
connectedHomeDebugAndroidTest - Installs and runs the tests for homeDebug on connected devices.
connectedSkeletonDebugAndroidTest - Installs and runs the tests for skeletonDebug on connected devices.
connectedUiDebugAndroidTest - Installs and runs the tests for uiDebug on connected devices.
deviceAndroidTest - Installs and runs instrumentation tests using all Device Providers.
deviceCheck - Runs all device checks using Device Providers and Test Servers.
lint - Runs lint on all variants.
lintBasicDebug - Runs lint on the BasicDebug build.
lintBasicRelease - Runs lint on the BasicRelease build.
lintDevDebug - Runs lint on the DevDebug build.
lintDevRelease - Runs lint on the DevRelease build.
lintHomeDebug - Runs lint on the HomeDebug build.
lintHomeRelease - Runs lint on the HomeRelease build.
lintSkeletonDebug - Runs lint on the SkeletonDebug build.
lintSkeletonRelease - Runs lint on the SkeletonRelease build.
lintUiDebug - Runs lint on the UiDebug build.
lintUiRelease - Runs lint on the UiRelease build.
lintVitalBasicRelease - Runs lint on just the fatal issues in the basicRelease build.
lintVitalDevRelease - Runs lint on just the fatal issues in the devRelease build.
lintVitalHomeRelease - Runs lint on just the fatal issues in the homeRelease build.
test - Run unit tests for all variants.
testBasicDebugUnitTest - Run unit tests for the basicDebug build.
testBasicReleaseUnitTest - Run unit tests for the basicRelease build.
testDevDebugUnitTest - Run unit tests for the devDebug build.
testDevReleaseUnitTest - Run unit tests for the devRelease build.
testHomeDebugUnitTest - Run unit tests for the homeDebug build.
testHomeReleaseUnitTest - Run unit tests for the homeRelease build.
testSkeletonDebugUnitTest - Run unit tests for the skeletonDebug build.
testSkeletonReleaseUnitTest - Run unit tests for the skeletonRelease build.
testUiDebugUnitTest - Run unit tests for the uiDebug build.
testUiReleaseUnitTest - Run unit tests for the uiRelease build.
To see all tasks and more detail, run gradlew tasks --all
To see more detail about a task, run gradlew help --task <task>
BUILD SUCCESSFUL in 1s
1 actionable task: 1 executed
I have no idea. I will try to reproduce this by cloning the repo on another machine. On what platform are you trying this? Do you have gradle installed? (gradle --version)
[fkooman@android-builder ~]$ cd Projects/eduvpn-android-20191126105955/
[fkooman@android-builder eduvpn-android-20191126105955]$ which gradle
/usr/bin/which: no gradle in (/home/fkooman/.local/bin:/home/fkooman/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin)
[fkooman@android-builder eduvpn-android-20191126105955]$ gradle --version
-bash: gradle: command not found
[fkooman@android-builder eduvpn-android-20191126105955]$
@fkooman From the set of tasks you see listed when you run ./gradlew tasks I think your environment needs to have ANDROID_HOME set as an environment variable pointing to your SDK dir. (I mentioned this in the long-running issue #206 particularly from https://github.com/eduvpn/android/issues/206#issuecomment-551069537 onwards)
Try setting it and then re-run the gradle tasks, good luck :)
@dzolnai thank you for the notes on how to run tests for Android apps. Can you elaborate on when you run which tests please? Also, do you run them on devices? AVD's? and if so, do you have a particular set of devices and/or AVDs you tend to use? I'd like to understand more of what you do and when so I can try to establish a reliable and trustworthy CI that runs appropriate tests at appropriate times. You're probably the best placed individual to explain how the app is currently tested from a development perspective :)
Does it even make sense for me to run the tests when building the app for production releases? It seems it is rebuilding everything (NDK?) just for the tests? How does that work?
Try setting it and then re-run the gradle tasks, good luck :)
Thanks! It works a bit better now :)
- What went wrong: Execution failed for task ':app:connectedBasicDebugAndroidTest'. com.android.builder.testing.api.DeviceException: No connected devices!
I guess this is where it ends for me ;-) Don't want to run the emulator on the build machine. Are there any tests that can be run without (emulated) devices?
@fkooman Generally, any tests that are in folders such as androidTest will require an Android runtime in order to run. Many projects create an emulated device (known as an AVD Android Virtual Device) as part of their CI build script. It's also possible to run them on real devices (generally but not exclusively connected by USB). There's also a framework called Roboelectric that aims to match the Android APIs that enables tests to run in a JVM on the build machine without a device. Roboelectric is impressive nonetheless it's not the real thing and people have mixed results using it, some love it and use it extensively, others avoid it (I tend to avoid it personally).
Can you elaborate on when you run which tests please?
I usually run all tests before I make a pull request.
Also, do you run them on devices? AVD's?
Mostly on device because I already have it attached, but sometimes on emulator too.
and if so, do you have a particular set of devices and/or AVDs you tend to use? I'd like to understand more of what you do and when so I can try to establish a reliable and trustworthy CI that runs appropriate tests at appropriate times. You're probably the best placed individual to explain how the app is currently tested from a development perspective :)
The tests should be successful on all devices that the app supports (so Android 5+). I recommend higher Android versions because they have added features (and with those, limitations), which enable for broader testing, such as dynamic permissions since Android 6, Doze since Android 7, or background execution restrictions since Android 8. Also app install speeds are a lot faster on later versions compared to Android 5, so your test will be done quicker too.
Are there any tests that can be run without (emulated) devices?
As of now, none. All tests depend on Android framework dependencies, such as SharedPreferences