Improvement: Allow possibility of treating java assertions as test Fails not Errors

[ktkopone]

This may just be a question of me not being able to find it in the docs/examples, but: In our use case we run the bulk of our testing in snippet RPC calls to the device, with java/JUnit assertions being raised within those calls. However, when an assertion fails in the RPC calls, an ApiError is instead propagated up to the python test (seemingly due to the RPC call crashing), and the test thus shows an 'error' rather than a fail. The same holds true for regular java AssertionError.

Since ApiError is propagated up, it isn't ~possible~ easy to use python-side code to distinguish proper test fails from errors, and the test summary will show 'errors' where 'fails' would be more desirable.

It would be nice if there were at least an arg option to make java/junit assertion fails get caught and propagated up through the rpc call into mobly's fail exception type or an assertionerror. This could similarly be done for junit skips.

Sample stacktrace:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/mobly/base_test.py", line 463, in exec_one_test
    test_method()
  File "app/src/snippet/moblyTests/app_tests.py", line 77, in test_sign_up
    self.deviceA.mbs.signUp()
  File "/usr/local/lib/python3.7/site-packages/mobly/controllers/android_device_lib/jsonrpc_client_base.py", line 336, in rpc_call
    return self._rpc(name, *args)
  File "/usr/local/lib/python3.7/site-packages/mobly/controllers/android_device_lib/jsonrpc_client_base.py", line 307, in _rpc
    raise ApiError(self._ad, result['error'])
mobly.controllers.android_device_lib.jsonrpc_client_base.ApiError: <AndroidDevice|owner> 
-------------- Java Stacktrace ---------------
junit.framework.AssertionFailedError: blah
	at junit.framework.Assert.fail(Assert.java:50)
	at <our package name>.SignUpActivityTest.signUp(SignUpActivityTest.java:76)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.google.android.mobly.snippet.manager.SnippetManager.invoke(SnippetManager.java:166)
	at com.google.android.mobly.snippet.manager.SnippetManager.invoke(SnippetManager.java:90)
	at com.google.android.mobly.snippet.rpc.MethodDescriptor.invoke(MethodDescriptor.java:85)
	at com.google.android.mobly.snippet.rpc.JsonRpcServer.handleRPCConnection(JsonRpcServer.java:118)
	at com.google.android.mobly.snippet.rpc.SimpleServer$ConnectionThread.run(SimpleServer.java:109)
----------------------------------------------

[xpconanfan]

Yeah, we've been thinking about this. It seems reasonable to propagate Junit's assertion errors as Mobly's assert errors. This would require changes on both snippet client and server sides though.

We can give this a try in Mobly 1.9

[xpconanfan]

Sorry this has been pushed back a lot. Because the scope needed to enable this is bigger than expected.

We are considering creating a generic mechanism to propagate errors from snippet server to client side. To make this non-trivial change to snippet lib, we need to first clean up some legacy issues, including the divergence in the implementation of snippet lib for various platforms.