RunFailed extension with sub tests

[roosht3]

What are you trying to achieve?

Trying to run this

 vendor/bin/codecept run --skip api --skip acceptance --no-interaction --no-exit

and get a failed file to tests/_output directory

What do you get instead?

I don't get the failed test in tests/_output instead I get it from the last run module. For example the file is inside Stock moduke

Stock/tests/_output/failed with main failed

/Users/xxx/AdminTest.php:testItCanFindRegisteredAdmin

Details

• Codeception version: 4.1
• PHP Version: 7.4
• Operating System:
• Installation type: Composer
• Suite configuration: Multiple suite with multi application

paths:
    tests: tests
    output: tests/_output
    data: tests/_data
    support: tests/_support
    envs: tests/_envs
actor_suffix: Tester
params:
    - .env.testing
extensions:
    enabled: [Codeception\Extension\RunFailed]
coverage:
    enabled: true
    include: ['Modules/*']
include:
    - 'Modules/*'
bootstrap: _bootstrap.php

[benlake]

Codeception version: 4.2.1

There is an incompatibility with the RunFailed extension and its inclusion via a suite configuration file. RunFailed triggers on the event "result.print.after", however, suite registered extensions are de-registered before that event; on the "suite.after" event. So while the extension is loaded/enabled/registered via inclusion in the suite's configuration, it never receives the event it is listening for.

I don't know if this is a bug or design incompatibility that needs to be surfaced; stating that some extensions are not compatible with being enabled via suite configuration.

The docs say extensions can be included in suite configurations, but based on the observed behavior, this can not be said generally and should likely have a qualifier concerning the events the extension supports. https://codeception.com/docs/08-Customization#Enabling-Extension

Extensions can also be enabled per suite inside suite configs (like acceptance.suite.yml) and for a specific environment.

A related oddity is that RunFailed is supposedly enabled by default in v2.1+, again according to current docs, but I've not seen that to be true (leaving no configuration for RunFailed does not yield a failed file); only specific inclusion in codeception.yml is functional. https://codeception.com/extensions#RunFailed

Starting from Codeception 2.1 this [RunFailed] extension is enabled by default.

[Naktibalda]

Starting from Codeception 2.1 this [RunFailed] extension is enabled by default.

It means that when you run codeception bootstrap, it generates codeception.yml file with enabled RunFailed extension.

[benlake]

It means that when you run codeception bootstrap, it generates codeception.yml file with enabled RunFailed extension.

Hmm, alright, I'm not sure that makes it across in the docs, but thank you for that clarification.