Specify a list of tests to run

[sebastianbergmann]

Implement a new CLI option --filter-using-test-id-file <file> to select the tests to be run based on a list of test identifiers that is read from a plain text file. This file has one test identifier per line.

[dafeder]

Is there some way this could support piping that list via stdin also? This would make integration with CI and other automation tools a lot simpler.

[dwidyna]

Are there any plans to finish this feature ? Or is there any other way to acomplish such filtering ?

I want to run test in batches (to run them in parallel in CI environment).

Upon PHPUnit 9 I was overwriting method TestCase::runTest to achieve this. Since PHPUnit 10 method run is final, so it cannot be overridden anymore.

    public function run(TestResult $result = null): TestResult
    {
        if ($this->shouldRun() === false) {
            return $result;
        }

        return parent::run($result);
    }

    protected function shouldRun(): bool
    {
        if ($this->runningTestsInBatchesIsEnabled() === false) {
            return true;
        }

        return $this->getIdOfBatchForThisTest() === $this->getIdOfBatchCurrentlyBeingRun();
    }

[dwidyna]

as a quick-fix for my challenge I moved shouldRun logic to setUp method

[dafeder]

@dwidyna I have set up parallel runs using groups. It's not merged yet but you can see the logic in my working branch here:

https://github.com/GetDKAN/dkan/blob/9626da08c77f5d1a3006d458975ce4c67ab74da6/.circleci/config.yml#L162

I assigned groups functional1, functional2, functional3 manually to my (slow) functional tests. So in my parallel runs, anything without one of those groups (my unit and kernel tests and any functional tests I forgot to tag) gets assigned to node 0, and functional[n] gets assigned to node n. My CI platform (circleCI) does track timing data that I'd rather use to automatically split my tests in a balanced way, but I've not really found a good way to pass an arbitrary list of tests to run. This feature would greatly help.

[dwidyna]

thanks for sharing your solution @dafeder I'd rather have something more random & dynamic. We run tests split into 32 batches. Splitting tests manually is far from ideal. So far we still use approach based on skipping the test, but it is not perfect neither...as now ~96% of tests in each batch are skipped, only ~3% (100% / 32 batches) are being executed

@sebastianbergmann any plans to implement such a filter ? Or maybe I'm missing some possibility that can be implemented using existing test runner ?

[dwidyna]

worth to mention, that for calculating the batch, I use full name of the test (with dataprovider part). This way allows better split amongst the batches:

    public function setUp(): void
    {
        if ($this->shouldRun() === false) {
            $this->markTestSkipped('Test is not part of this batch');
        }
    }

    private function shouldRun(): bool
    {
        if ($this->runningTestsInBatchesIsEnabled() === false) {
            return true;
        }

        return $this->getIdOfBatchForThisTest() === $this->getIdOfBatchCurrentlyBeingRun();
    }

    protected function getIdOfBatchForThisTest(): int
    {
        if ($this->runningTestsInBatchesIsEnabled() === false) {
            throw new \RuntimeException('Running test in batches is not enabled in configuration');
        }

        $hashForThisTest = crc32($this->toString());

        $remainder = $hashForThisTest % $this->getNumberOfBatches(); // 0 or more

        $idOfBatchForThisTest = $remainder + 1;

        return $idOfBatchForThisTest;
    }

[dwidyna]

I guess that ability to inject own/custom Filter would be most beneficial in this case: https://github.com/sebastianbergmann/phpunit/blob/988d49128ae098ab5bf0b4a05f66d7b3246ecef5/src/TextUI/TestSuiteFilterProcessor.php#L24