RuleTestCase: add analyseFileWithErrorsAsComments

[janedbal]

trafficstars

I'd like to suggest this feature I'm using for about 4-5 years in my projects. E.g. here.

Benefits:

• immediately visible where the error is raised and where not
• errors float with code, so you can easily add code in between existing errors without modifying expected errors (currently even a single use statement added breaks everything)

Downsides:

• no support for multiple errors on a single line
• when traits come into play, error lines are not always in ascending order so it mismatches

---

I believe that at least 90% tests could benefit from this. Other can stick with the old approach.

[phpstan-bot]

You've opened the pull request against the latest branch 1.11.x. If your code is relevant on 1.10.x and you want it to be released sooner, please rebase your pull request and change its target to 1.10.x.

[DaveLiddament]

I've created a similar package to do help with testing custom rules: https://github.com/DaveLiddament/phpstan-rule-test-helper. This provides AbstractRuleTestCase to extend for testing custom rules.

This allows you to either hard code the error message in the fixture file (similar to @janedbal's suggestion) . E.g.

    $item->updateName("world"); // ERROR can not call method

---

If the same error message is repeated multiple times, then you can also override the getErrorFormatter with the error message. This means error messages need updating only in one place in the test:

  protected function getErrorFormatter(): string 
  {
    return "Can not call method";
  } 

The fixture simplifies to:

    $item->updateName("world"); // ERROR 

---

Finally you can provide context in the error messages:

  protected function getErrorFormatter(): string 
  {
    return "Can not call method {0} from {1}";
  } 

The fixture error has additional contextual information, if there is more than 1 bit it is separated by | character.

    $item->updateName("world"); // ERROR Item::updateName|Foo

In the above example the expected error message would be: Can not call method Item::updateName from Foo

---

It be nice to either bring similar functionality into PHPStan, or highlight helper packages that can help with testing.

[ondrejmirtes]

Some thoughts after letting this sit for a while:

• Expecting errors in comments is very similar to ignoring errors
• If we implement a similar system I want it to work like assertType - that it can be tested both by running PHPUnit and simply by analysing code with PHPStan
• So what about @phpstan-error An exact error message that works the same way as @phpstan-ignore with an error identifier in 1.11.x?

[DaveLiddament]

Some thoughts after letting this sit for a while:

• Expecting errors in comments is very similar to ignoring errors
• If we implement a similar system I want it to work like assertType - that it can be tested both by running PHPUnit and simply by analysing code with PHPStan
• So what about @phpstan-error An exact error message that works the same way as @phpstan-ignore with an error identifier in 1.11.x?

I like @phpstan-error.

I love the fact you'd then not even need to create PHPUnit tests, you can just point PHPStan at the code snippet.

[janedbal]

So what about @phpstan-error An exact error message that works the same way as @phpstan-ignore with an error identifier in 1.11.x?

I'm not sure that the next-line magic is needed here. Imo not.

[janedbal]

For the meantime, we open-sourced our solution to a standalone package with just this functionality (inline error asserts in tested code): https://github.com/shipmonk-rnd/phpstan-dev