Wrong error sometimes matched when errors are similar

[Zeckie]

Describe the problem, ie expected/actual result (if it's not blatantly obvious)

When there are changes near baselined errors, the current algorithm used to match errors to the baseline sometimes ignores the wrong error, resulting in both false positives and false negatives. Similar issues occur if changes (additions or deletions) are made that result in offsets changing more than 50 lines. Troubleshooting these issues currently require knowledge of the rules used to match errors.

There is also a real risk that when one error is fixed and another error is introduced (that is similar to the fixed one, such as having the same code and message), that basedmypy will incorrectly treat it as being the same error.

Steps to reproduce

I encountered this issue in a real project, where the errors reported were in a function about 50 lines after where I was making changes. I used --write-baseline, so I could see the changes it made to the baseline file, and compare to the changes I made to the code. The new errors written to the baseline had the same values for message and code as the errors they incorrectly matched. In the example below, the errors are quite obvious.

before baseline

def foo() -> None:
    a: int = "not an int"


def bar() -> None:
    b: int = "not an int"

baseline file

{
  "files": {
    "example.py": [
      {
        "code": "assignment",
        "column": 13,
        "message": "Incompatible types in assignment (expression has type \"str\", variable has type \"int\")",
        "offset": 2,
        "target": "example.foo"
      },
      {
        "code": "assignment",
        "column": 13,
        "message": "Incompatible types in assignment (expression has type \"str\", variable has type \"int\")",
        "offset": 4,
        "target": "example.bar"
      }
    ]
  },
  "format": "1.3",
  "targets": [
    "file:."
  ]
}

code changed after baseline

def foo() -> None:
    a: int = "not an int"


def baz() -> None:
    # optional comment so error on different line
    c: int = "newly added, not an int"


def bar() -> None:
    b: int = "not an int"

mypy output

example.py:11:14: error: Incompatible types in assignment (expression has type "str", variable has type "int")  [assignment]
        b: int = "not an int"
                 ^
Found 1 error in 1 file (checked 1 source file)

Potential solution

TLDR: Detect that this issue has likely occurred, and give user more detail to make it easier to resolve

If any of the following occur, disable automatic baseline update and print message indicating a potential mismatch:

• there are multiple errors that could match a baselined one, and either there are more errors detected in the code than in the baseline, or the distance between them has changed
• a new previous/next error is similar that would have been considered a match if it had a different offset
• changes are to multiple fields (such as both target and offset)
• offset has changed more than 50 lines (currently results in it not being considered a match, but could just be some new, valid code added between baselined errors)

Some details that could be useful in the displayed message are:

• the details usually displayed for the error
• line numbers of previous / next baselined errors (to help troubleshoot issues where blocks of code have been added)
• which fields matched / didn't match

Given the limited details currently stored in the baseline, there will be some "grey areas" where basedmypy cannot tell whether an error is the one in the baseline or not. It may need options to be able to tune it to make these checks more or less sensitive.

Basedmypy version

1.3.1 and 1.4.0rc1

Python version used

3.10

Operating system and version

Windows