Failed test can't be re runned if the failed test use Codeception/Specify

[moltam]

What are you trying to achieve?

Re run the failed tests:

./codecept run -g failed

What do you get instead?

Failed tests that use Codeception/Specify won't run. This problem only affects these tests.

How to reproduce

Create test file:

class FailedTest extends \Codeception\Test\Unit
{
    use \Codeception\Specify;

    public function testWithSpecify()
    {
        $this->specify('Specification', function () {
            $this->assertFalse(true, 'Message 1');
        });
    }

    public function testWithoutSpecify()
    {
        $this->assertFalse(true, 'Message 2');
    }
}

Run tests:

$ ./codecept run
Codeception PHP Testing Framework v2.2.3
Powered by PHPUnit 5.4.8 by Sebastian Bergmann and contributors.

Acceptance Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Functional Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Unit Tests (2) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
✖ FailedTest: With specify | specification | examples index 0 
✔ FailedTest: With specify (0.01s)
✖ FailedTest: Without specify (0.00s)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 167 ms, Memory: 8.00MB

There were 2 failures:

---------
1) FailedTest: With specify | specification | examples index 0
 Test  tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0
Message 1
Failed asserting that true is false.
Codeception/Codeception#1  /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:11
Codeception/Codeception#2  FailedTest->{closure}
Codeception/Codeception#3  /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:12
Codeception/Codeception#4  FailedTest->testWithSpecify

---------
2) FailedTest: Without specify
 Test  tests/unit/FailedTest.php:testWithoutSpecify
Message 2
Failed asserting that true is false.
Codeception/Codeception#1  /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:17
Codeception/Codeception#2  FailedTest->testWithoutSpecify

FAILURES!
Tests: 2, Assertions: 2, Failures: 2.

_output/failed now contains the tests:

tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0
tests/unit/FailedTest.php:testWithoutSpecify

Run failed tests:

$ ./codecept run -g failed
Codeception PHP Testing Framework v2.2.3
Powered by PHPUnit 5.4.8 by Sebastian Bergmann and contributors.
[Groups] failed 

Acceptance Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Functional Tests (0) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Unit Tests (1) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
✖ FailedTest: Without specify (0.01s)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------


Time: 195 ms, Memory: 8.00MB

There was 1 failure:

---------
1) FailedTest: Without specify
 Test  tests/unit/FailedTest.php:testWithoutSpecify
Message 2
Failed asserting that true is false.
Codeception/Codeception#1  /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:17
Codeception/Codeception#2  FailedTest->testWithoutSpecify

FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

Note: Only testWithoutSpecify runned.

Possible solution

I did some digging, and maybe I found the problem. I think that in Lib\GroupManager an strpos() call's parameters are swapped.

The $testPattern contains: /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:testWithSpecify | Specification | examples index 0. And $filename . ':' . $test->getName(false) is: /mnt/store-ssd/work/codeception/tests/unit/FailedTest.php:testWithSpecify.

So the strpos() returns false. But if I swap the two parameters, the all the failed tests are re runned.

I am new to Codeception, so I might be wrong (I haven't set up the tests, so I can't verify that this change doesn't break anything).

Details

• Codeception version: 2.2.3
• PHP Version: 7.0.8
• Operating System: Ubuntu 16.04.1 LTS
• Installation type: Composer
• List of installed packages (composer show)

behat/gherkin                      v4.4.1 Gherkin DSL parser for PHP 5.3
codeception/codeception            2.2.3  BDD-style testing framework
codeception/specify                0.4.3  BDD code blocks for PHPUnit and Codeception
doctrine/instantiator              1.0.5  A small, lightweight utility to instantiate objects in PHP without invoking their constructors
facebook/webdriver                 1.1.2  A PHP client for WebDriver
guzzlehttp/guzzle                  6.2.1  Guzzle is a PHP HTTP client library
guzzlehttp/promises                1.2.0  Guzzle promises library
guzzlehttp/psr7                    1.3.1  PSR-7 message implementation
myclabs/deep-copy                  1.5.1  Create deep copies (clones) of your objects
phpdocumentor/reflection-common    1.0    Common reflection classes used by phpdocumentor to reflect the code structure
phpdocumentor/reflection-docblock  3.1.0  With this component, a library can provide support for annotations via DocBlocks or otherwise retrieve information that is embedded in a DocBlock.
phpdocumentor/type-resolver        0.2    
phpspec/prophecy                   v1.6.1 Highly opinionated mocking framework for PHP 5.3+
phpunit/php-code-coverage          4.0.1  Library that provides collection, processing, and rendering functionality for PHP code coverage information.
phpunit/php-file-iterator          1.4.1  FilterIterator implementation that filters files based on a list of suffixes.
phpunit/php-text-template          1.2.1  Simple template engine.
phpunit/php-timer                  1.0.8  Utility class for timing
phpunit/php-token-stream           1.4.8  Wrapper around PHP's tokenizer extension.
phpunit/phpunit                    5.4.8  The PHP Unit Testing framework.
phpunit/phpunit-mock-objects       3.2.3  Mock Object library for PHPUnit
psr/http-message                   1.0    Common interface for HTTP messages
sebastian/code-unit-reverse-lookup 1.0.0  Looks up which function or method a line of code belongs to
sebastian/comparator               1.2.0  Provides the functionality to compare PHP values for equality
sebastian/diff                     1.4.1  Diff implementation
sebastian/environment              1.3.7  Provides functionality to handle HHVM/PHP environments
sebastian/exporter                 1.2.2  Provides the functionality to export PHP variables for visualization
sebastian/global-state             1.1.1  Snapshotting of global state
sebastian/object-enumerator        1.0.0  Traverses array structures and object graphs to enumerate all referenced objects
sebastian/recursion-context        1.0.2  Provides functionality to recursively process PHP variables
sebastian/resource-operations      1.0.0  Provides a list of PHP built-in functions that operate on resources
sebastian/version                  2.0.0  Library that helps with managing the version number of Git-hosted PHP projects
symfony/browser-kit                v3.1.3 Symfony BrowserKit Component
symfony/console                    v3.1.3 Symfony Console Component
symfony/css-selector               v3.1.3 Symfony CssSelector Component
symfony/dom-crawler                v3.1.3 Symfony DomCrawler Component
symfony/event-dispatcher           v3.1.3 Symfony EventDispatcher Component
symfony/finder                     v3.1.3 Symfony Finder Component
symfony/polyfill-mbstring          v1.2.0 Symfony polyfill for the Mbstring extension
symfony/yaml                       v3.1.3 Symfony Yaml Component
webmozart/assert                   1.0.2  Assertions to validate method input/output with nice error messages.

• Suite configuration:

I have set up a new project to demonstrate this. I've installed Codeception and Specify with composer (composer require codeception/codeception codeception/specify), and I've run bootstrap: ./codecept bootstrap.

So, there is nothing special, default installation:

actor: Tester
paths:
    tests: tests
    log: tests/_output
    data: tests/_data
    support: tests/_support
    envs: tests/_envs
settings:
    bootstrap: _bootstrap.php
    colors: true
    memory_limit: 1024M
extensions:
    enabled:
        - Codeception\Extension\RunFailed
modules:
    config:
        Db:
            dsn: ''
            user: ''
            password: ''
            dump: tests/_data/dump.sql

# Codeception Test Suite Configuration
#
# Suite for unit (internal) tests.

class_name: UnitTester
modules:
    enabled:
        - Asserts
        - \Helper\Unit

[moltam]

Maybe related: Codeception/Codeception#3390 and Codeception/Codeception#3155

[DavertMik]

Yes, they seem to be related. RunFailed extension is not updated to handle examples and specify test names.

[Naktibalda]

The root cause here is that Specify creates test instances during execution, not when the tests are loaded, so there is nothing to filter.