phpunit icon indicating copy to clipboard operation
phpunit copied to clipboard

Fixed missing the assert diff message of phpt in a teamcity output

Open benbor opened this issue 4 years ago • 3 comments

Hi!

  1. I've added missed diff output in case of failure phpt test in --teamcity configuration, when phpt scenario contains --EXPECT-- section.
  2. I've put the diff output in case of failure phpt test with section --EXPECTF-- into the same style as --EXPECT--.

Background

PHPUnit 9.5, failed test scenarious in *.phpt files, output in --teamcity Note: --teamcity output is also used by PHPStorm, so all screenshot examples are taken from it.

image

As you can see:

  1. --EXPECT-- and --EXPECTF-- output is different.
  2. <Click to see difference> tool is absent

First attempt

In the first attempt https://github.com/sebastianbergmann/phpunit/commit/1086fedd4ea3df6ba6320a7d93cd33afecff4b2b I connected missed --EXPECT-- diff to the message, in the same style, as it had been done for --EXPECTF--

Pros:

  • That were small code changes

Cons:

  • Diff was printed as plained text, i.e. the tool <Click to see difference> still hadn't work

First attempt result, the <Click to see difference> was absent:

image

Second attempt (current version)

I founded, that PHPStorm can work with diff in prettier way (you can see how common PHPUnit\Framework\TestCase works on top of screenshots), and I started to investigate. I've discovered the following:

  • PHPTAssertionFailedError is not consistent with ExpectationFailedException, which works perfect in PHPUnit\Framework\TestCase test cases.
  • PHPTAssertionFailedError displayed information in message for --EXPECTF--, because of https://github.com/sebastianbergmann/phpunit/pull/3659
  • Difference in --EXPECT-- and --EXPECTF-- exists because phpunit uses different constraints: StringMatchesFormatDescription and IsEqual. The first one used it's own attach diff in message implementation.

So, I fixed all of them together, because they are coupled. What has been done:

  1. Now StringMatchesFormatDescription works with expected diff as other classes: manipulates with ComparisonFailure object. I used the same idea of implementation as in JsonMatches constraint.
  2. Interface ComparisonFailureContains has been created, to have possibility to accept ComparisonFailure object from both classes: PHPTAssertionFailedError and ExpectationFailedException
  3. Added two end-to-end logger tests for both --EXPECT-- and --EXPECTF-- cases.

In comparison with First attempt fix:

Pros:

  • both phpt scenarios --EXPECT-- and --EXPECTF-- have <Click to see difference>
  • framework class src/Framework/TestFailure.php became independed from PHPTAssertionFailedError
  • StringMatchesFormatDescription works with ComparisonFailure as all others.

Cons:

  • More changes

Second attempt result:

image

Questions to maintainers:

  • Please, let me know which solution is okay for PHPUnit, the First or the Second.
  • Please, let me know whether I should do additional patches? And for which branches? This is 9.5 branch. Are patches in 8.5 enough? Or master also is required? Sorry, it is my very first PR in this repo, so I can't figure out whether this pr is a bugfix or a new feature

Thanks!

benbor avatar Nov 25 '21 09:11 benbor

@sebastianbergmann Hi! May I wonder, do I violated some contribution guidelines or everything is ok from my side, and I just need wait some time? Thanks!

benbor avatar Dec 07 '21 14:12 benbor

I did not have time to look into this yet, sorry.

sebastianbergmann avatar Dec 07 '21 15:12 sebastianbergmann

Codecov Report

Merging #4831 (44313bf) into 9.5 (6bcc8e6) will increase coverage by 0.17%. The diff coverage is 100.00%.

Impacted file tree graph

@@             Coverage Diff              @@
##                9.5    #4831      +/-   ##
============================================
+ Coverage     83.84%   84.02%   +0.17%     
+ Complexity     4633     4632       -1     
============================================
  Files           274      274              
  Lines         11972    11972              
============================================
+ Hits          10038    10059      +21     
+ Misses         1934     1913      -21     
Impacted Files Coverage Δ
...Framework/Exception/ExpectationFailedException.php 100.00% <ø> (ø)
src/Runner/PhptTestCase.php 68.82% <ø> (+1.08%) :arrow_up:
...nstraint/String/StringMatchesFormatDescription.php 100.00% <100.00%> (ø)
...c/Framework/Exception/PHPTAssertionFailedError.php 100.00% <100.00%> (ø)
src/Framework/TestFailure.php 100.00% <100.00%> (ø)
src/Util/Log/TeamCity.php 71.73% <100.00%> (+13.04%) :arrow_up:
src/Framework/Constraint/Constraint.php 97.36% <0.00%> (-2.64%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 6bcc8e6...44313bf. Read the comment docs.

codecov[bot] avatar Dec 07 '21 15:12 codecov[bot]

@sebastianbergmann sorry for ping, I talked with you about this PR on Symfony conf in Disney :) It would be very kind if you found time to answer two questions

  1. Please, let me know which solution is okay for PHPUnit, the First or the Second.
  2. Please, let me know whether I should do additional patches? And for which branches? This is 9.5 branch. Are patches in 8.5 enough? Or master also is required? Sorry, it is my very first PR in this repo, so I can't figure out whether this pr is a bugfix or a new feature

Then, if this PR is actual, I help with refactoring for master and release it in Phpunit 10 Thanks!

benbor avatar Dec 05 '22 21:12 benbor