BetterReflection icon indicating copy to clipboard operation
BetterReflection copied to clipboard

[Demo] Use Infection with native PHPStan integration

Open maks-rafalko opened this issue 5 months ago • 4 comments

This is not intended to be merged (at least for now), more like to have a discussion and to compare with https://github.com/Roave/BetterReflection/pull/1505


This PR:

  • uses Infection with native integration of PHPStan. Now, we can just run infection --static-analysis-tool=phpstan and it will do the job. So it's opt-in, by default PHPStan is not used
  • what is the difference between native integration and tools like roave/infection-static-analysis-plugin/ and phpstan/mutant-killer-infection-runner? Infection runs PHPStan processes in parallel (depending on --threads) while those plugins run Psalm/PHPStan processes sequentially because of the nature of the the "hack" used there.

There is still too much work to do, here are the major issues for now:

  • it's slow. For some (not yet known) reason PHPStan process takes at least 1s even for 1-file project
  • we don't use --stop-on-first-error from PHPStan (because it doesn't exist yet xD)
  • many of the Mutants are killed by PHPStan when they shouldn't

For example:

@@ @@
             if ($reflection instanceof ReflectionFunction && $this->containsLine($reflection, $lineNumber)) {
                 return $reflection;
             }
-            if ($reflection instanceof ReflectionConstant && $this->containsLine($reflection, $lineNumber)) {
+            if (true && $this->containsLine($reflection, $lineNumber)) {
                 return $reflection;
             }
         }

$ '/app/vendor/bin/phpstan' '--tmp-file=/tmp/infection/mutant.a99f5bbe9bbbb1514da2bdbe5e315d5c.infection.php' '--instead-of=/app/src/Util/FindReflectionOnLine.php' '--error-format=json' '--no-progress' '-vv'
  
PHPStan error: Left side of && is always true.

^ PHPStan always kills this Mutant, but in reality the tests can easily have a whole here, not testing properly the left side of this condition. I think on Infection side we need to mark some mutators (mutator operators) to not require static analysis, and just finish if they are escaped, not wasting time with static analysis.

cc @ondrejmirtes @staabm

maks-rafalko avatar Jun 03 '25 22:06 maks-rafalko

Please run GH actions. (working example is on my fork: https://github.com/maks-rafalko/BetterReflection/pull/2)

maks-rafalko avatar Jun 03 '25 22:06 maks-rafalko

GH actions are not executed automatically :(

Now (among tens of other improvements) the output shows killed mutants by PHPStan by marking them with A so it looks like this:

[...]

.................A.............AAM.A....A.........   (3650 / 4409)
.......A......M.MM.AA..A.......A...A.........A....   (3700 / 4409)
......M.AAA...A.................A.A.M..MA....A....   (3750 / 4409)
.................A.......A.....................M..   (3800 / 4409)
.AA...A..A...............A.AAA......A............E   (3850 / 4409)
E..A..........A...................................   (3900 / 4409)
..........................A.......................   (3950 / 4409)
..............................T...................   (4000 / 4409)
..................................................   (4050 / 4409)
.........T........................................   (4100 / 4409)
..................................................   (4150 / 4409)
T..............E..................AA..............   (4200 / 4409)
.........M................................A.......   (4250 / 4409)
.....................................A..........E.   (4300 / 4409)
E.E.............................................AA   (4350 / 4409)
....................A.............A...............   (4400 / 4409)
......A.A                                            (4409 / 4409)

4409 mutations were generated:
    3660 mutants were killed by Test Framework
+     212 mutants were killed by Static Analysis
       0 mutants were configured to be ignored
      11 mutants were not covered by tests
      47 covered mutants were not detected
       7 errors were encountered
       0 syntax errors were encountered
       9 time outs were encountered
     463 mutants required more time than configured

Metrics:
         Mutation Score Indicator (MSI): 98%
         Mutation Code Coverage: 99%
         Covered Code MSI: 98%

maks-rafalko avatar Jun 10 '25 16:06 maks-rafalko

GH actions are not executed automatically :(

as long as the PR has a merge conflict with the target branch no github actions will start

staabm avatar Jun 10 '25 16:06 staabm

looks like it requires it since I'm not a contributor

maks-rafalko avatar Jun 10 '25 16:06 maks-rafalko