[Feature Request] Custom Assert Failure Messages

[denil-ct]

What's the issue?

When an assert statement fails, its quite hard to see what went wrong. This is especially worsened in the case of junit reports where its very hard to find what check went wrong. For example, if you have a JS script that evaluates something (let's say it makes an API call to a local server) and returns a bool. Now when I assert if the value is indeed true, I get a vague error like this, which makes it hard to track what went wrong.

Assertion is false: false is true

Proposed solution

A custom descriptive assert message that gets emitted to the logs and the junit reports which helps us pinpoint the exact assert statement that went wrong and why? For example XCTest has a similar feature where you can add a descriptive message as the last parameter for all the XCTAssert methods.

XCTAssertTrue(isSuccessful, "isSuccessful should be true in this case as the API status is 200")

Similarly the assert syntax can be modified to accommodate this change. Something like this perhaps

- assertTrue:
    condition: ${output.result}
    message: "The feed API fetch failed" #The message parameter can be optional as well.

Alternatives considered

Printing log messages to console works to an extent, but that seems to be missing from the final junit reports that are generated.

[Fishbowler]

Thought I'd check how this works with the new labels functionality. It's not better.

appId: com.example
---
- assertTrue:
    condition: ${1==2}
    label: "1 is the same as 2"

gives

Running on iPhone 14 Pro - iOS 16.4 - 1FCE17AC-A5FF-496C-9EDB-7C66E9D0FBE3      
                                                                                
 ║                                                                              
 ║  > Flow                                                                      
 ║                                                                              
 ║    ❌  1 is the same as 2                                                     
 ║                                                                              
                                                                                
Assertion is false: false is true

[Fishbowler]

The text is being generated from here: https://github.com/mobile-dev-inc/maestro/blob/319e46626e97e85be04c81d6b095aa47e37dcab4/maestro-orchestra/src/main/java/maestro/orchestra/Orchestra.kt#L317-L320

with the second half coming from here: https://github.com/mobile-dev-inc/maestro/blob/a72aa5717c616009fdc3ef0c5bab05d83872f36d/maestro-orchestra-models/src/main/java/maestro/orchestra/Condition.kt#L42-L44

@denil-ct Would you replace both with a single message, or would it make sense to replace just the second half?

[denil-ct]

@Fishbowler did not know a similar thing was already in the works. Nice work. I tried running from main, but unfortunately couldn't get it to work. I copy pasted the same flow as yours, but got this error.

Could not parse Flow file:

Cannot deserialize value of type `java.lang.String` from Object value (token `JsonToken.START_OBJECT`)
 at [Source: UNKNOWN; byte offset: #UNKNOWN] (through reference chain: java.util.ArrayList[0]->maestro.orchestra.yaml.YamlFluentCommand["assertTrue"])

Would you replace both with a single message, or would it make sense to replace just the second half?

I would say just the second part. The first part can act as the title and the second part can be a detailed description of what went wrong.

Also, quick question if the junit flag is enabled, does the xml file still have the label or does it just have the Assertion is false... line?

[Fishbowler]

I've not had use for the junit part yet. I've got no idea how it works there.

Definitely feels like the label should be separate from the error text. e.g.

appId: com.example
---
- assertTrue:
    condition: ${1==2}
    label: "Check that 1 is the same as 2"
    errorText: "1 and 2 were different"

Last time I had problems running main was after the project dropped IDB - scrubbing build artefacts and starting fresh sorted that.

[denil-ct]

I am okay with either way. The benefit of splitting is each command is more focused which also has a side effect of being verbose which can be good/bad depending on who you ask.

I think I'll side with the split approach a bit more than the other.

Main compiles and runs fine for me, its just that when I modify the assert command to include a label it stops working.

[Fishbowler]

The label stuff was only merged very recently. Pulled latest?

The docs for it describe a bit about why it exists: https://github.com/mobile-dev-inc/maestro-docs/pull/25

Some assertions and waits and taps can look vague - sometimes something a little longer can be more descriptive and maintainable to Future Us.

If I get some time tomorrow, I'll take a look at whether a PR for custom error text like this would be doable without enormous changes that would require a keener eye from Maestro folk.

[denil-ct]

Ah yes, my bad. I did take a pull from main, but the main was from my fork, which did not have the upstream changes 🤦

If I get some time tomorrow, I'll take a look at whether a PR for custom error text like this would be doable without enormous changes that would require a keener eye from Maestro folk.

Sure, let me know if I can help in any manner.

[Fishbowler]

@igorsmotto Before I go wading in, can I check that something like this is likely to be accepted?

- assertTrue:
    condition: ${output.result}
    errorText: "The feed API fetch failed"

[Wei18]

Assertion is false: false is true On junit report. It's difficulty to identify the root cause, please let us add more message for all of Assert APIs.