RFE: Suggestions for improving the usability of the CN1 TestRecorder

[ThomasH99]

Is your feature request related to a problem? Please describe. Automating tests are critical for any serious app development. But it can also be annoying and boring so good tool support is essential to make it happen :-). And could be great selling point for CN1 :-) There's lot of work already done in the Test Builder, but after having tried to use it for real, there are several shortcomings for 'industrial' use.

Describe the solution you'd like I've filed some issues on this already (here and here), but here are the highlights (there are probably more things since these issues blocked me from getting started):

• Generating a test interactively is great, but it should be easy to edit the test code as it's generated. E.g. when you add Asserts on Labels, you typically get too many, and it is essential to be able to directly delete checks on fields or values that are irrelevant to your test as they are generated (directly the TestRecorder UI) and while you have the UI visible in the simulator.
• Generating Asserts when you have a list of things in the UI will generate Asserts for alle displayed elements, most of which is irrelevant. It would be useful to somehow select the UI elements for which to generate the Asserts.
• Structure tests, e.g. have a base case that will set up the test state, e.g. move to the relevant screen in which to perform multiple tests. I guess this could be handled by referring to existing test cases from inside another test case.
• Make it easy to describe/document the test cases (in addition to the Test Name) by adding free text comments directly in the UI.
• It would be particular great if you could start building a skeleton test plan with test names and descriptions and then gradually add the meat (actual tests) to them. This might also enable outsourcing the detailed creation of test cases.
• Right now, fields etc are often identified by the text inside them, but that is likely to change and break all tests. Being able to refer to named fields would be great.
• Not sure about this, but it seems that if you have a list of elements, and then click a button in one them, the TestRecorder will always pick the first one (and not the one clicked during simulation).
• Nicely integrated into the new Simulator UI. Would be really cool to have it turning in the background and save any manually executed test case 'after the fact'.

Describe alternatives you've considered I don't think there's any alternative (other than manually testing your app).

[shai-almog]

I looked into these and a lot of these are a bit problematic. Editing the source code would require deep integration with the IDE since the source code isn't generated at this time and editing capabilities requires a far more complex model than we currently have.

We have buttons to generate various types of asserts at a given point.

Structuring tests would require the recorder to know/parse other tests which is pretty hard.

The list of elements issue seems to be a bug. If so it's a separate issue and requires a test case.

I don't understand the last suggestion.

[ThomasH99]

Thanks for looking at this. By "Editing the source code" I simply meant to allow editing (in particular deleting) the generated test code shown in the 'Test Recorder' bottom-right window. I.e. the test code starting with

import com.codename1.testing.AbstractTest;

import com.codename1.ui.Display;

public class UnnamedTest extends AbstractTest {
    public boolean runTest() throws Exception {

I am aware of the "buttons to generate various types of assert" like "Title", "Labels" and "Text Areas" but they typically generate way too many asserts since they generate for everything shown in the app UI, where you typically only want to generate for the elements that are relevant for your particular test case. Hence the need mentioned above to support deleting the unneeded test code.

But an even better solution would probably be to build on the new Simulator "Inspect Component" and eg right-click whatever app UI elements you want to test and have a context-menu with Labels/Text Areas buttons to generate asserts just for the selected UI component.

I don't have much QA experience with the typical ways of structuring test suites (e.g. what is implemented by tools like the old HP Quality Center). But in practice, maybe it would be enough to add a left-hand project hierarchy browser showing existing test packages and test cases, and then you can click any to run it as part of the new test case you're working on. E.g. if you create a new test case "Delete item" you'd click the existing test case "Add item" to run the test case that creates the item you're going to test deleting.

The last suggestion was that the Test Recorder might be worth integrating into the Simulator UI. You could have an option where it runs and always record your manual interaction with the app. Then whenever you've done an interactive simulation and has come across an error situation, you would already have the test commands (e.g. "executeToolbarCommandAtOffset(1); clickButtonByName("ItemListEditButton");") recorded and can easily reuse them to rerun the same test steps automatically. E.g. you test your app, navigate to a screen, select an option and notice that a label is not correctly updated. Then to create a test case to run after you've fixed the label update issue, you could simply copy the steps (navigate to screen, select option) into a test case, and click assert on the label, and correct the assertLabel to test for the correct value of the label. I hope this explains the idea better?