false positive syntax errors when using syntax for newer python version in `.pyi` files

[DetachHead]

Summary

# foo.pyi

def foo[T](): ...

type Foo = int

Cannot use type parameter lists on Python 3.8 (syntax was added in Python 3.12)
Cannot use `type` alias statement on Python 3.8 (syntax was added in Python 3.12)

since .pyi files are not executed at runtime, they should be able to use newer syntax than the target python version

Version

0.14.6

[MichaReiser]

Thanks for reporting. Yes, that makes sense. @ntBre I think this would be great to fix unless a contributor gets to it first :)

One option is to ensure we always pass the latest Python version to parse if the file is a pyi file, or we add a new option to ParseOptions to signify whether we're parsing a stub file (requires reviewing all call sites)

[11happy]

I have linked the PR fixing it. Thank you : )

[AlexWaygood]

since .pyi files are not executed at runtime, they should be able to use newer syntax than the target python version

while this is true, the most popular type checker (mypy) currently uses the Python ast module to parse all Python files, including .pyi files. If mypy is invoked using Python 3.11, it won't be able to parse syntax introduced in Python 3.12. So if a .pyi file uses Python 3.12+ syntax, it won't be usable with mypy unless you use Python 3.12 or newer to invoke mypy. I don't think that's what most authors of stubs would want: I think most authors of stubs would want their stubs to be legible by all type checkers, regardless of what Python version you use to invoke that type checker.

[DetachHead]

the way i see it, mypy's days as "the most popular type checker" are numbered, so ruff should not be held back by it. ruff should be able to support projects that don't care about supporting mypy anymore. i would think any project that cares about mypy support on older python versions versions would already be running mypy with that version of python in their CI anyway, so i don't think it's ruff's responsibility to account for this. but i guess this behavior could be configurable if this is really a concern?

[AlexWaygood]

As a developer of ty, I obviously hope that ty succeeds :-)

And as a developer of typeshed, I would of course love to be able to use the latest and greatest Python syntax when writing stubs!

But as an author of Python libraries, and a developer of typeshed, I would very much want any stubs I write to work with whatever type checker a user of my library might be using. Whether you're a fan of mypy or not, the fact of the matter is that for many users that type checker is currently mypy, and it will continue to be mypy for many users for many years to come, even if it loses its spot as the most popular type checker.

There has been talk over at mypy of switching to a new parser (possibly the Ruff parser!), so it would be interesting to revisit this discussion if and when that happens. But unless and until that happens, I would definitely at least want to be able to opt into these syntax errors on .pyi files (and making them opt-out makes most sense IMHO).

[MichaReiser]

(and making them opt-out makes most sense IMHO)

which they already are. Or more specifically, you can opt-out of the syntax errors by using per-file-target-version and setting it to 3.14 for all pyi files

[flying-sheep]

A few points:

• I don’t know if that only applies to ty, but at least ty already unconditionally allows UnionType syntax: https://github.com/astral-sh/ty/issues/1525
• ty doesn’t have a per-file-target-version setting, right?

I’m personally interested in making the following .pyi file content parse without either Ruff or ty complaining in a project with requires-python = '>= 3.12':

class Spam[E = Eggs]: ...

[AlexWaygood]

at least ty already unconditionally allows UnionType syntax: PEP 604 union syntax should be allowed in stubs (.pyi) on all Python versions ty#1525

That's somewhat different because it can be parsed by the Python ast module on all versions of Python. It's not actually new syntax, formally speaking (it fails at runtime on older Python versions, but the CPython parser can parse it on older Python versions)

[flying-sheep]

That makes sense. So Ruff has a workaround, how to make this work in ty?

[AlexWaygood]

So Ruff has a workaround, how to make this work in ty?

You're correct that ty doesn't have a per-file-target-version setting, and it wouldn't be realistic to implement such a setting for a multifile analysis tool like a type checker.

It would be nice if it was possible to suppress syntax errors -- then you could just add a per-file-ignore for this specific syntax error for all .pyi files, if you're happy to accept that your stub files are likely not to work with certain type checkers.

[MichaReiser]

I'd prefer an option to either disable this behavior or changing the syntax version per file over making syntax errors suppressable

[AlexWaygood]

I'd be okay with an allow-new-stub-syntax = true configuration setting for ty. But I still don't really understand the opposition to making some syntax errors suppressable. I think we've seen in several user reports at this point that users expect these errors to be suppressable just like any other Ruff/ty diagnostic, and it seems like a much more generalised, versatile solution than introducing new configuration settings just for this. I don't understand how it harms the user experience in any way to make them suppressable, even if it's not something that most users should ideally ever need.

Anyway, I don't feel that strongly. I'm okay with a new configuration setting for ty if you think that's the way to go here.

[MichaReiser]

We've only had very few reports and some of them were related to bugs in the parser.

It's also easier to have a single config flag than to configure a handful of rules for what should be allowed in the pyi files.

[flying-sheep]

Question is if there is any other use case where different subsets of a project's files are supposed to be parsed with different settings and then typechecked together.

Apart from monorepos I can't think of anything, and there, file subsets are subprojects, so parsing is entirely defined by requires-python.

[AlexWaygood]

I think it's reasonably common to have a scripts/ directory that requires a newer Python version to the "production" directories elsewhere in your project. But I would generally just invoke the type checker on those directories separately. If you're using ty as an LSP, I guess you'd want something like https://github.com/astral-sh/ty/issues/1554? But I wouldn't usually have a separate pyproject.toml file in my scripts/ directory

[flying-sheep]

Right, instead of a pyproject.toml, scripts would have an inline script metadata block that could define the requires-python.

I haven’t checked how monorepo support works for uv/ruff. Is it just adding more pyproject.toml files? If so, inline script metadata should work the same!