Maestro CLI: Running specific test AND excluding relevant tags still runs the test

[matt-dalton]

Is there an existing issue for this?

• [x] I have searched the existing issues and didn't find mine.

Steps to reproduce

Possibly this is specific to our use-case, but feels like wrong functionality regardless.

In CI, we want to run all our tests across 3 separate machines in parallel. The sharding arguments I think don't work because they require multiple devices running on the same machine.

To work around this, we have a script that collates all the tests, splits them up, and then runs the ones relevant for that node. So we end up with a command like this:

for TEST_FILE in $TEST_FILES; do
  echo "Running test: $TEST_FILE"
  if ! maestro test --config=e2e/config.yml "$TEST_FILE" "@.env.maestro.$PLATFORM" --env MAESTRO_APP_ID="$MAESTRO_APP_ID" --exclude-tags="$EXCLUDE_TAGS"; then
    EXIT_CODE=1
    break
  fi
  echo "Completed test: $TEST_FILE"
done

The problem I'm now having is I want to add tags to exclude certain tests - e.g. exclude android-only from ios runs.

If I run a specific test while excluding a tag within that test - e.g.

maestro test --config=e2e/config.yml "e2e/tests/my-android-only-test.yml" "@.env.maestro.ios" --env MAESTRO_APP_ID="xx.xx.xx" --exclude-tags="android-only"

It'll still try and run the test.

I can see how this would occur, but it seems "more correct" to me to exclude the tags even if you're running a specific test.

Actual results

The test attempts to run, even though the exclude-tags say it shouldn't

Expected results

It should say 'No tests found' or something

About app

Closed source React Native app Version here doesn't matter

About environment

Java 17 macOS 15.7.2 (arm64)

Logs

Logs

Running on iPhone 16 Pro - iOS 18.6 - 561E09CC-6AD7-4CD3-AE0E-60367778F620                                                                                                                             
   
 ║                                                                                                                                                                                                     
 ║  > Flow: ......

Maestro version

2.0.9

How did you install Maestro?

install script (https://get.maestro.mobile.dev)

Anything else?

No response

[linear[bot]]

MAE-345

[Fishbowler]

There's some fairly out-of-the-ordinary usage in your example.

Could I simplify this down to something like:

flow.yaml

appId: com.example
tags:
  - do_not_run
---
- launchApp

Command:

maestro test --exclude-tags do_not_run some_flow.yaml

You expect it to run nothing, but here, flow would run?

Or it there more to it?

[matt-dalton]

There's some fairly out-of-the-ordinary usage in your example. This is true, although not sure how else to achieve what I was trying to do (running tests across multiple machines)

You've summarised the issue there yes.

[Fishbowler]

Right now, the inverse (using includes) works:

maestro test --include-tags non-existent .

You'll get some red output

Include / Exclude tags did not match any Flows:

Include Tags:
- non-existent

Exclude Tags:

This feels like the appropriate outcome for this issue too.

But... You also get a non-zero exit status. I think I'd expect this to stay consistent too. But that might difficult for your use case, where you won't be able to use exit status to distinguish between no flows and failed flows.

[matt-dalton]

But that might difficult for your use case, where you won't be able to use exit status to distinguish between no flows and failed flows. Well...I guess worst case you could parse the error message. Or is there a specific exit status that might help catch/narrow it down?