Better testing of platform releases regarding plugin compatibility

[janpio]

The 7.1.3 release of cordova-android had a breaking bug with multiple plugins that included app in their name, including our own cordova-plugin-inappbrowser.

The cordova-android tests did not catch that.

We should figure out a way to better test our platforms with plugins.

[janpio]

The plugin tests (run on CI via paramedic) themselves would probably have caught that, as they create a project and then see if functionality of the plugins work. But those tests are done with the released Cordova CLI and platforms.

We have the mobilespec project, but I am not exactly sure what that is and how it works.

[brody4hire]

I just raised https://github.com/apache/cordova-mobile-spec/pull/147 to document a few known quirks & issues:

• npm install inside cordova-js is needed to resolve a local grunt issue
• A Gradle file in platforms/android subdirectory tries to read non-existing debug keys. Workaround is to comment the lines out of the Gradle file.
• It may be required to uninstall the cordova-plugin-compat plugin due to multiple "BuildHelper" classes defined in dex files (https://stackoverflow.com/questions/46562289/multiple-dex-files-define-lorg-apache-cordova-buildhelper/46562523#46562523)

[brody4hire]

and not all of the automatic tests are passing for me. It seems to help to select 1 plugin at a time to run the tests on.

[janpio]

So mobilespec is one thing.

~~For Android I just discovered this: https://github.com/apache/cordova-android/tree/master/test (Was linked at https://github.com/apache/cordova-coho/blob/master/docs/platforms-release-process.md#android-extras, under https://github.com/apache/cordova-coho/blob/master/docs/platforms-release-process.md#ios-extras mentions something similar for iOS)~~

[brody4hire]

So mobilespec is one thing.

For Android I just discovered this: [...] ([...] mentions something similar for iOS)

Here are the updated Android & iOS unit test links for the sake of clarity, in response to https://github.com/apache/cordova/issues/54#issuecomment-441290852 above:

• cordova-android / test (unchanged) linked in: https://github.com/apache/cordova-coho/blob/master/docs/platforms-release-process.md#extra-android-unit-tests
• cordova-ios / tests / CordovaLibTests linked in: https://github.com/apache/cordova-coho/blob/master/docs/platforms-release-process.md#extra-ios-unit-tests

NOTE that these unit tests are included by npm test and are run by CI, according to https://github.com/apache/cordova/issues/54#issuecomment-442457744 below.

[janpio]

How is this related to plugins @brodybits?

(All those tests are include in npm test and thus run by CI, the Android ones for example are run via npm run java-unit-tests)

[brody4hire]

How is this related to plugins @brodybits?

Here is my response, along with a motion below.

I provided updated information in response to some information with an outdated link in https://github.com/apache/cordova/issues/54#issuecomment-441290852. I edited my response to clarify.

(All those tests are include in npm test and thus run by CI, the Android ones for example are run via npm run java-unit-tests)

Thanks for the info, which triggered https://github.com/apache/cordova-coho/pull/213. I added the info to my response above.

I would like to motion the following:

• no more comments about unit tests in this issue; https://github.com/apache/cordova-coho/pull/213 is the conclusion so far; any more comments should be in https://github.com/apache/cordova-coho/pull/213 or in a new issue
• we hide this comment and the 3 comments above (https://github.com/apache/cordova/issues/54#issuecomment-441290852, https://github.com/apache/cordova/issues/54#issuecomment-442449615, and https://github.com/apache/cordova/issues/54#issuecomment-442457744) as "off topic", "resolved", or whatever else makes sense
• this motion should be seconded by actually hiding the comments and not adding any new comments

My motion is amended as follows:

• https://github.com/apache/cordova/issues/54#issuecomment-441290852 is not to be hidden but strikeout is to be applied to the part about Android and iOS unit tests. This amendment is seconded by action already taken.

And I am doing one more thing as a collaborator: mark the comments hidden so far as "resolved". My rationale is that the first hidden comment I wrote was in response to a comment by someone else (fair discussion) before we decided to apply the strikeout.

[brody4hire]

So mobilespec is one thing.

This and https://github.com/apache/cordova-mobile-spec/issues/184#issuecomment-442827057 imply that there are some other ways to test platform releases and maybe some other package releases that I don't know about. Can you enlighten me and others?

[janpio]

So mobilespec is one thing.

This

That was the conclusion of us finding and understanding mobilespec as a way to test plugins compatibility on platforms in the comments before (which this topic is about).

apache/cordova-mobile-spec#184 (comment)

The comment you link to is about plugin testing, not platform releases. Generally plugin testing is done via all the plugin tests, that are run via cordova-paramedic in CI. But that doesn't offer any (simple) options to control the platform being used.

But if you have such questions, please ask them in a new issue, Slack or dev mailing list thread and don't abuse one of my issues that has a clear topic or question - which is not about enlightening you.

[janpio]

Step back:

What would help to "test platform releases regarding plugin compatibility"? How can we find out if a new platform release breaks anything with plugins?

1. Check if plugins install on the platform
2. Check if normal app (without plugin usage) still works on platform with plugin installed
3. Check if plugin tests app (cdvtests/index.html via cordova-plugin-test-framework) still works on platform with plugin installed
4. Check if plugin Appium tests (currently only runnable on CI via cordova-paramedic) still work on platform with plugin installed

Other ideas?

[janpio]

Could we maybe somehow compare what a plugin installation does between two platform versions? That could catch if changes in our "copy files from a to b" logic (that caused problems in the past) will break a plugin without executing its code.

[brody4hire]

I will need some time to evaluate these ideas. I think more comprehensive testing would be good on major releases and occasional minor releases. I would really favor a more straightforward testing procedure for hotfixes and other patch releases.

[janpio]

This exists:

https://github.com/alsorokin/cordova-periodic-build-android https://github.com/alsorokin/cordova-periodic-build-ios

(Take a look at the Travis build history)

Broken right now, but could be a blueprint on how to create a repo that can test platforms with all plugins.