steward
steward copied to clipboard
Filter option can break tests dependency tree
When option --filter is used and it filter out all but one testcase, which has dependency on some previous one (which hasn't been run), the testcase is run despite of this (and may fail on some precondition).
For example, when run in
$ vendor/bin/steward run production firefox -vvv --filter DelayedExample
...
My\DelayedExampleTest> There was 1 error:
My\DelayedExampleTest>
My\DelayedExampleTest> 1) My\DelayedExampleTest::testShouldReturnSearchResults
My\DelayedExampleTest> Lmc\Steward\Component\LegacyException: Cannot read Legacy file /.../steward-example/selenium-tests/logs/seed-data.legacy
My\DelayedExampleTest>
My\DelayedExampleTest> /.../steward-example/selenium-tests/vendor/lmc/steward/src/Component/Legacy.php:200
My\DelayedExampleTest> /.../steward-example/selenium-tests/tests/DelayedExampleTest.php:32
Maybe tests with "No tests executed" should be marked as failed / or skipped? Or the --filter option could only be allowed for test methods, not for classes (because for classes the --pattern option should be used).
Maybe the value passed to PHPUnit --filter option could always be prefixed with something like ::, to make sure the filtering is only applied to test methods, not test classes. See https://phpunit.de/manual/5.7/en/textui.html#textui.examples.filter-patterns
However, this is only first part of the solution.