steward icon indicating copy to clipboard operation
steward copied to clipboard

Filter option can break tests dependency tree

Open OndraM opened this issue 8 years ago • 1 comments
trafficstars

When option --filter is used and it filter out all but one testcase, which has dependency on some previous one (which hasn't been run), the testcase is run despite of this (and may fail on some precondition).

For example, when run in

$ vendor/bin/steward run production firefox -vvv --filter DelayedExample
...
My\DelayedExampleTest> There was 1 error:
My\DelayedExampleTest> 
My\DelayedExampleTest> 1) My\DelayedExampleTest::testShouldReturnSearchResults
My\DelayedExampleTest> Lmc\Steward\Component\LegacyException: Cannot read Legacy file /.../steward-example/selenium-tests/logs/seed-data.legacy
My\DelayedExampleTest> 
My\DelayedExampleTest> /.../steward-example/selenium-tests/vendor/lmc/steward/src/Component/Legacy.php:200
My\DelayedExampleTest> /.../steward-example/selenium-tests/tests/DelayedExampleTest.php:32

Maybe tests with "No tests executed" should be marked as failed / or skipped? Or the --filter option could only be allowed for test methods, not for classes (because for classes the --pattern option should be used).

OndraM avatar Feb 14 '17 22:02 OndraM

Maybe the value passed to PHPUnit --filter option could always be prefixed with something like ::, to make sure the filtering is only applied to test methods, not test classes. See https://phpunit.de/manual/5.7/en/textui.html#textui.examples.filter-patterns

However, this is only first part of the solution.

OndraM avatar Feb 14 '17 22:02 OndraM