xharness "Select All Mac tests" execute non-Mac tests

[spouliot]

Steps to Reproduce

1. cd tests
2. make runner
3. Click Select then All Mac tests
4. Click Execute then Run all selected tests

Expected Behavior

Only runs macOS related tests.

Actual Behavior

Run some iOS and tvOS tests, e.g. monotouch-test, which makes the run longer and noisier.

Environment

xcode12.2 Have not tried main

Build Logs

https://gist.github.com/spouliot/77af9f27f4e510aedf22e8bca55db725

The gist shows failure to run monotouch-test which should not have been executed.

[rolfbjarne]

Unfortunately the test selection implementation is not very intuitive: "Select" will only add to the selection, not take away from it. Coupled with the fact that there's a default set of enabled tests, means you have to:

1. Deselect all tests
2. Select all Mac tests

To only run Mac tests.

[spouliot]

That rings a bell... I likely forgot it. Tried it and it still queued some non-mac tests (but less than before)

dont link (Mac Modern)
introspection (Mac Modern)
xammac tests (Mac Modern)
xammac tests (Mac Modern)
xammac tests (Mac Modern)
xammac tests (Mac Modern)
link all (Mac Modern)
link all (Mac Modern)
link sdk (Mac Modern)
link sdk (Mac Modern)
mono-native-compat (Mac Modern)
mono-native-unified (Mac Modern)
[NUnit] Mono Mac OS X BCL tests group 1 (Mac Full)
[NUnit] Mono Mac OS X BCL tests group 2 (Mac Full)
[NUnit] Mono Mac OS X BCL tests group 3 (Mac Full)
[xUnit] Mono Mac OS X BCL tests group 4 (Mac Full)
mscorlib (Mac Full)
[NUnit] Mono Mac OS X BCL tests group 1 (Mac Modern)
[NUnit] Mono Mac OS X BCL tests group 2 (Mac Modern)
[NUnit] Mono Mac OS X BCL tests group 3 (Mac Modern)
[xUnit] Mono Mac OS X BCL tests group 4 (Mac Modern)
mscorlib (Mac Modern)
apitest (Mac Full)
dont link (Mac Full)
dont link (Mac System)
mono-native-compat (Mac Full)
mono-native-unified (Mac Full)
Xtro (Mac)
introspection (tvOS - simulator)
introspection (watchOS 32-bits - simulator)
mmptest (macOS)

[spouliot]

so it looks like it's all the oldest simulators that were executed - which have iirc a different logic applied.

[spouliot]

I did not notice it earlier but after clicking "Deselect" and "All tests" this is what xharness shows

3 tests' device not found, 0 tests passed

Trying to deselect device tests does not change the result. Those 3 tests seems un-de-select-able :)

[spouliot]

Funny it worked as expected with xcode12.2 on 10.15.7 (Intel) Just not on 11.0 beta 10 (Apple)

[spouliot]

Original state is

Test run in progress (NotStarted: 55, Ignored: 1200, DeviceNotFound: 3)

and the DeviceNotFound (actually simulator not found) are the one executed