Wrong test extension linked in consecutive tests

[andreaswolf]

What are you trying to achieve?

I have the following setup:

• two functional test cases
  • each has its own test extension in a subfolder Fixtures/test_extension with a matching ext_emconf.php and composer.json
  • the composer.json does not have a package name (if this is not supported anymore, I will add it)
• the test extensions are symlinked via $testExtensionsToLoad

What do you get instead?

When executing the test cases individually, everything works. When executing both in one run (with an empty var/tests/ folder), the test extension from the first test case is symlinked to both test systems.

How to reproduce the issue?

See above.

I have an example in the branch "broken-extension-linking" of my EXT:migrations: https://github.com/andreaswolf/typo3-ext-migrations/tree/broken-extension-linking

Additional information you would like to provide?

I think the problem is using static info inside \TYPO3\TestingFramework\Composer\ComposerPackageManager.

Specify some data of the environment

• TYPO3 testing framework version: 7.0.2+ (it works in 7.0.0 and 7.0.1, the commit that breaks it is af280b6c34f4c17cec3e8f12acb25665fb0ae504.
• TYPO3 version: 11.5
• TYPO3 installation type: Composer
• PHP version: 7.4, 8.0, 8.1
• Web server used + version: n/a
• Operating system: Ubuntu 22.04

[DanielSiepmann]

We run into the same issue. They are not cleaned up between runs. The corresponding setting seems to be expected to be TestClass instead of test case specific.

Our workaround is to write and call this method after parent::setUp() (Mind the hardcoded path which needs adjustment):

    /**
     * The testing framework doesn't expect the list to change between tests in a single class.
     *
     * We therefore build our own setup to change extensions for each test.
     */
    private function setUpExtensionsToLoad(): void
    {
        $testExtensions = $this->testExtensionsToLoad;
        array_shift($testExtensions);

        foreach ($testExtensions as $testExtension) {
            $path = $this->getInstancePath() . '/typo3conf/ext/' . basename($testExtension);
            unlink($path);

            $source = __DIR__ . str_replace('typo3conf/ext/e2_core/Tests/Functional/Service', '', $testExtension);
            $success = @symlink($source, $path);
            if (!$success) {
                throw new Exception(
                    'Can not link the path ' . $source . ' to ' . $path,
                    1389969623
                );
            }
        }
    }

[sbuerk]

Test fixture extension must be unique regarding composer.json name and extension key.

Note that this would also fail later when moving to a composer based install using a central registration as multiple path extension could not share the same name extension key.

Consider that similar to have two or more local path folders for extensions/packages and having two extension in differnt folders with the same name (composer.json/extension-key).

I was never happy with the base name lookup but it was the only way to make the prior used typo3conf/ext/<extension_key> notation working in some way. It's quite nasty to get all running in some way and badly mangling here now again would not fix it when composer based functional test instance may be introcued at some point.

Better make test fixture extensions unique.

Forgot that I wanted to add a easy way to pre-register test-fixture extensions (which needs to be unique anyway) with some kind of path/path globs so they could be required simply by the unique composer package name then (like any extension/core extension required/required-dev in the root composer.json.

Thus not adding the resymlink approach because the "old" information would be stalled in the static package information cache anyway (clearing it would make the whole thing slower on executiong gathering all the information for each run again).

Will have a look again, but for now I would suggest to make them unique (a good thing anyway to avoid confusion).

[lolli42]

Hey. We looked at this again: Mitigating this issue is quite ugly, and we postulate that extension keys - even for test fixture extensions - need to be unique. We will close this issue as won't fix.

[d-s-codappix]

Well the issue is not having different extensions with same name, but in general you can't change the extensions between tests within the same class. Our concrete use case: We have a Service that adds page specific TypoScript from Filesystem to TYPO3. Basically replacing sys-template on page level with file system. This is often necessary on large legacy systems where you wanna get rid of TypoScript within Database.

We test this service with a single TestClass with multiple test cases. The following is part of our setUp() method:

        $this->testExtensionsToLoad[] = 'typo3conf/ext/e2_core/Tests/Functional/Service/TypoScriptServiceTest/Extensions/' . ucfirst((string) $this->name()) . '/test_ext';
        $this->testExtensionsToLoad[] = 'typo3conf/ext/e2_core/Tests/Functional/Service/TypoScriptServiceTest/Extensions/' . ucfirst((string) $this->name()) . '/test_ext0';
        $this->testExtensionsToLoad[] = 'typo3conf/ext/e2_core/Tests/Functional/Service/TypoScriptServiceTest/Extensions/' . ucfirst((string) $this->name()) . '/test_ext1';

We could change it to something like

        $this->testExtensionsToLoad[] = 'typo3conf/ext/e2_core/Tests/Functional/Service/TypoScriptServiceTest/Extensions/' . ucfirst((string) $this->name()) . '_test_ext';
        $this->testExtensionsToLoad[] = 'typo3conf/ext/e2_core/Tests/Functional/Service/TypoScriptServiceTest/Extensions/' . ucfirst((string) $this->name()) . '_test_ext0';
        $this->testExtensionsToLoad[] = 'typo3conf/ext/e2_core/Tests/Functional/Service/TypoScriptServiceTest/Extensions/' . ucfirst((string) $this->name()) . '_test_ext1';

But that wouldn't help due to https://github.com/TYPO3/testing-framework/blob/main/Classes/Core/Functional/FunctionalTestCase.php#L267 only initializing the testExtensionsToLoad on first test within the class.

I just wanted to explain that in case you missunderstood the scenario. I, of course, will follow your decision. We will either keep our workaround in place as long as possible, or switch to multiple test classes extending a base class. I assume there are only a few cases where one needs such a setup.

[andreaswolf]

@d-s-codappix In my case it was actually two separate test files, so that's not like your case. I knew about the one-time initialization of the test extensions (for performance reasons; the list is also not re-initialized for a second test run within five (?) minutes after the first run, since the extension directories already exist).