combine `--list-tests` with `--group`

[staabm]

running php vendor/bin/phpunit --group levels --list-tests on the PHPStan-src codebase returns the same list of tests, as when not using --group levels: php vendor/bin/phpunit --list-tests

I would expect --group levels to apply the same filtering as then running phpunit itself with --group levels.

$ vendor/bin/phpunit --version
PHPUnit 9.5.23 #StandWithUkraine

$ php -v
PHP 8.2.12 (cli) (built: Oct 24 2023 21:15:35) (NTS Visual C++ 2019 x64)
Copyright (c) The PHP Group
Zend Engine v4.2.12, Copyright (c) Zend Technologies

what I want to achieve: I want to use phpunit test filtering capabilities to separate the huge PHPStan test-suite into groups which I can run in parallel in GitHub Actions. running it in several parallel GitHub Action jobs has the benefit of utilizing more CPUs and therefore a faster feedback loop

[staabm]

asked differently: maybe there is already a mechanism to determine a list of test-classes given a group-name?

[staabm]

just leaving a note for myself: the initial --list-tests feature was implemented in https://github.com/sebastianbergmann/phpunit/commit/ebc228036d0fd06d2d48b080657af2684634f7ee which can be taken as a source of inspiration to implement the request

[mfn]

I want to use phpunit test filtering capabilities to separate the huge PHPStan test-suite into groups which I can run in parallel in GitHub Actions.

It's not the same but for the same reason, I tried to make something like https://github.com/sebastianbergmann/phpunit/pull/4449, for when paratest can't be used.

Grouping is a very manual / tedious process, I wanted something easier: give me all tests, split them up (aka: slice them) and feed a slice of test back to be run by phpunit.

The PR didn't make it and, before that, I'm having some glue script which essentially take the output of --list-tests-xml and creates deterministic slices from which I create a phpunit.xml for each slice in the matrix.

Works "production grade" but I wish all that gluing wouldn't be necessary.

[staabm]

Grouping is a very manual / tedious process, I wanted something easier: give me all tests, split them up (aka: slice them) and feed a slice of test back to be run by phpunit.

implemented a similar thing for PHPStan in https://github.com/phpstan/phpstan-src/pull/2916 which allows parallelizing in a GitHub Action. (as no matter we will implement something in PHPUnit 11, it will take years before PHPStan can update to such a new PHPunit version ;))

[ondrejmirtes]

it will take years before PHPStan can update to such a new PHPunit version

We run the expensive levels tests only for a single PHP version which can be the latest one, we can update PHPUnit in CI before running the (sliced) tests.

[staabm]

@ondrejmirtes I am not sure whether its a good idea to run a portion of tests with a different PHPunit version, since it will emit different warnings/deprecations and it might be cumbersome to maintain tests in a way which works on several phpunit major versions

[ondrejmirtes]

@staabm We can definitely make it work ;)

[sebastianbergmann]

I would expect --group levels to apply the same filtering as then running phpunit itself with --group levels

Which is why newer versions of PHPUnit emit a warning when --group and --list-tests are used together, for example.

The intent for options such as --list-tests has always been to list all tests without applying a filter. Changing that could be considered a breaking change.

For a use case such as what you have in mind, I prefer to do filtering/sorting/partitioning of tests outside of PHPUnit. First, I use the --list-tests-xml option to generate a machine-readable list of tests:

ExampleTest.php

<?php declare(strict_types=1);
namespace example;

use PHPUnit\Framework\Attributes\Group;
use PHPUnit\Framework\TestCase;

#[Group('my-group')]
final class ExampleTest extends TestCase
{
    public function testOne(): void
    {
        $this->assertTrue(true);
    }
}

$ phpunit --list-tests-xml tests.xml ExampleTest.php
PHPUnit 11.0.3 by Sebastian Bergmann and contributors.

Wrote list of tests that would have been run to tests.xml

tests.xml

<?xml version="1.0"?>
<testSuite xmlns="https://xml.phpunit.de/testSuite">
 <tests>
  <testClass name="example\ExampleTest" file="/home/sb/ExampleTest.php">
   <testMethod id="example\ExampleTest::testOne" name="testOne"/>
  </testClass>
 </tests>
 <groups>
  <group name="my-group">
   <test id="example\ExampleTest::testOne"/>
  </group>
 </groups>
</testSuite>

Outside of PHPUnit, I can now use a script that loads tests.xml and performs the filtering/sorting/partitioning I need.

I do see value in what you propose, in fact, I think it makes a lot of sense. However, I do not know whether applying --filter, --group etc. when using --list-tests etc. would be a backward compatibility break or not and if so, how we best deal with it.

[staabm]

I did my initial tests with phpunit 9.x therefore did not realize there is also a --list-tests-xml variant, which contains everything we need.

I would therefore close this issue here.

I do see value in what you propose, in fact, I think it makes a lot of sense. However, I do not know whether applying --filter, --group etc. when using --list-tests etc. would be a backward compatibility break or not and if so, how we best deal with it.

if you still want to accept the PR, I am fine with finishing it of course. IMO adding a option to a command which was not supported before is not a BC break, but of course you have to decide on this one.

[sebastianbergmann]

Superseded by #5720.