Implement filtering tests based on XML input file

[mfn]

Summary

Export via --list-tests-xml and cut and slice as wanted, to feed it back via --tests-xml. Works with PHPUnit tests as well as PHPT tests.

A cheap slicing script is provided for now at https://gist.github.com/mfn/256636242cbe8a51252ce28181a6b074

Example:

~/src/phpunit $ ./phpunit --list-tests-xml tests.xml
PHPUnit 9.4-g567e55e11 by Sebastian Bergmann and contributors.

Wrote list of tests that would have been run to tests.xml

~/src/phpunit $ ./phpunit_xml_slicer.php tests.xml 1/100 > partial_tests.xml

~/src/phpunit $ ./phpunit --tests-xml partial_tests.xml
PHPUnit 9.4-g567e55e11 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.4.8
Configuration: /Users/neo/src/phpunit/phpunit.xml

.............................                                     29 / 29 (100%)

Time: 00:00.186, Memory: 22.00 MB

OK (29 tests, 29 assertions)

Overview

• Add a new --tests-xml <xml file> argument to PHPUnit
• Implement new filter \PHPUnit\Runner\Filter\XmlTestsIterator which parses the XML and builds an internal data structure to identify what tests to accept()

I consider the state of the PR in a "functional state" and before finishing it I would seek feedback if I missed parts and guidance what kind of tests are best.

Considerations

• ~The handling of using SimpleXML and parsing in \PHPUnit\Runner\Filter\XmlTestsIterator::setFilter is pretty crude. I barely worked with XML and PHP in the last decade. I saw \PHPUnit\Util\Xml\Loader but wasn't sure if it should be used, it was just added a few days ago. Also I liked the "simplicity" of SimpleXml to get this up and running.~ => Replaced with ext-xml aka DomDocument and friends
• [x] I noticed that the filters (not just the new one I added) are instantiated multiple times, thus the XML is parsed many more times. No idea what/if to do here
• I first wanted to name it --filter-tests-xml, seemed more apt. But I ran into a strange problem with sebastian/cli-parser I didn't really understand so I just gave it a different to get the ball rolling

Error when using --filter-tests-xml as option name

Testing started at 08:07 ...
/usr/local/bin/php /Users/user/src/phpunit/phpunit --configuration /Users/user/src/phpunit/phpunit.xml --teamcity
PHPUnit 9.4-g4719a8977 by Sebastian Bergmann and contributors.

Runtime:       PHP 7.4.8
Configuration: /Users/user/src/phpunit/phpunit.xml


Failed asserting that string matches format description.
--- Expected
+++ Actual
@@ @@
-PHPUnit %s by Sebastian Bergmann and contributors.
+string(11) "-tests-xml="
+int(31)
+int(87)
+string(6) "filter"
+string(7) "filter="
 
-...                                                                 3 / 3 (100%)
+Fatal error: Uncaught RuntimeException: dafuq in /Users/user/src/phpunit/vendor/sebastian/cli-parser/src/exceptions/AmbiguousOptionException.php:19
+Stack trace:
+#0 /Users/user/src/phpunit/vendor/sebastian/cli-parser/src/Parser.php(165): SebastianBergmann\CliParser\AmbiguousOptionException->__construct('--filter')
+#1 /Users/user/src/phpunit/vendor/sebastian/cli-parser/src/Parser.php(81): SebastianBergmann\CliParser\Parser->parseLongOption('filter', Array, Array, Array)
+#2 /Users/user/src/phpunit/src/TextUI/CliArguments/Builder.php(128): SebastianBergmann\CliParser\Parser->parse(Array, 'd:c:hv', Array)
+#3 /Users/user/src/phpunit/src/TextUI/Command.php(223): PHPUnit\TextUI\CliArguments\Builder->fromParameters(Array, Array)
+#4 /Users/user/src/phpunit/src/TextUI/Command.php(115): PHPUnit\TextUI\Command->handleArguments(Array)
+#5 /Users/user/src/phpunit/src/TextUI/Command.php(100): PHPUnit\TextUI\Command->run(Array, true)
+#6 Standard input code(9): PHPUnit\TextUI\Command::main()
+#7 {main}
 
-Time: %s, Memory: %s
-
-OK (3 tests, 3 assertions)
+Next PHPUnit\TextUI\Exception: dafuq in /User in /Users/user/src/phpunit/src/TextUI/Command.php on line 102

 /Users/user/src/phpunit/tests/end-to-end/filter-class-isolation.phpt:14
 /Users/user/src/phpunit/src/Framework/TestSuite.php:665
 /Users/user/src/phpunit/src/Framework/TestSuite.php:665
 /Users/user/src/phpunit/src/TextUI/TestRunner.php:668
 /Users/user/src/phpunit/src/TextUI/Command.php:147
 /Users/user/src/phpunit/src/TextUI/Command.php:100

TODO

• [x] Write tests; but what kind of? PHPT? Not sure where to put files / fixtures, so many already there. Guidance appreciated! => a phpt test has been added
• [x] XML parsing? Improve? Change to DomDocument? => Indeed, changed to DomDocument

Links

• "Specify a list of tests to run" => https://github.com/sebastianbergmann/phpunit/issues/3387

[codecov[bot]]

Codecov Report

Merging #4449 (1e80f8a) into master (ba39893) will increase coverage by 0.09%. The diff coverage is 90.47%.

@@             Coverage Diff              @@
##             master    #4449      +/-   ##
============================================
+ Coverage     83.54%   83.63%   +0.09%     
- Complexity     4554     4578      +24     
============================================
  Files           286      287       +1     
  Lines         11759    11820      +61     
============================================
+ Hits           9824     9886      +62     
+ Misses         1935     1934       -1     

Impacted Files	Coverage Δ	Complexity Δ
src/TextUI/Help.php	100.00% <ø> (ø)	23.00 <0.00> (ø)
src/TextUI/CliArguments/Configuration.php	68.58% <87.50%> (+0.19%)	269.00 <4.00> (+3.00)
src/Runner/Filter/XmlTestsIterator.php	88.37% <88.37%> (ø)	17.00 <17.00> (?)
src/TextUI/CliArguments/Builder.php	80.56% <100.00%> (+0.18%)	120.00 <0.00> (+1.00)
src/TextUI/CliArguments/Mapper.php	79.51% <100.00%> (+0.24%)	82.00 <0.00> (+1.00)
src/TextUI/TestRunner.php	65.71% <100.00%> (+1.30%)	219.00 <0.00> (+2.00)
src/Runner/DefaultTestResultCache.php	93.65% <0.00%> (+1.58%)	27.00% <0.00%> (ø%)

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update ba39893...1e80f8a. Read the comment docs.

[theseer]

Can we please stop using none namespaced XML and provide XSDs for them?

[mfn]

ping, anyone?

[theseer]

Not sure what you're expecting at this moment.

While the current state of this PR is having conflicts that should get resolved, my comment regarding the lack of a namespace and an XSD is unanswered. I do realize you're only relying on the export that already is broken in that regard. But before building more things that rely on this, we should fix that, imho.

[epdenouden]

@mfn I am working on writing a more detailed review. Long story very short: the idea of filtering tests via an XML-file and the main Iterator is fine, but it needs refactoring.

I am also wary of any iterator work getting in the way of the coming redesign of the core iterator/filtering mechanisms. If this PR gets accepted as-is, I would have to refactor this new functionality, too.

[mfn]

For some reason I never got the email notification from the review feedback https://github.com/sebastianbergmann/phpunit/pull/4449#pullrequestreview-507261805 , I just discovered it now 💥

I'll try to find time ASAP to address things!

[mfn]

Btw, is it fine to target master or should I target another branch, perhaps 9.5?

[mfn]

Is there anything else I can do? AFAIK I addresses the feedback or did I miss anything, which blocks us?

[maks-rafalko]

Let me up this ticket. Would like to help (if any help is needed) because this feature will greatly help us at Infection with filtering what tests to run (including data sets)

[theseer]

I didn't look into the Code itself, as @epdenouden already did that and I trust he'll review the current state as well ;)

But I'd like to repeat my statement made earlier: We should clean up the XML produced before we add more functionality and have (more) external projects rely on it. At the very least, let us please add a namespace with a version identifier so any future upgrades and changes are not going to kill us.

[mfn]

I definitely feel caught in a catch-22 here and I get no clear indicator how I can move forward.

I feels out of scope for me to tackle the "XML / namespace / schema" problem here, that's not a problem I created and seems "higher up" to me.

Randomly I tried to update the PR and resolve the merge conflicts, but I got demotivated due to lack of feedback / specific actionable stuff, so I didn't touch it for some time.

---

In any way, I'm about to leave for vacation and can't provide updates within the next few weeks. I'll check the PR feedback when I get back but I'm also happy if someone else steps up and assists here 🙏

[sebastianbergmann]

Thank you for your contribution. I appreciate the time you invested in preparing this pull request. However, I have decided not to merge it.