Investigate options to backport AST changes

[cdce8p]

If possible, it could be useful to add a custom AST parser which backports changes from newer versions.

The ast module is inherently unstable. We have already added a lot of code dealing with inconsistencies between Python versions but still find new / unexpected once. A custom AST parser which uses the latest "rules" could simplify the tree build in astroid while also providing the possibility to backport new features to older versions.

This is meant as a longshot idea. It might very well not be worth it in the end.

Some ideas

• Backport end_lineno and end_column_offset. This might depend on C API which would complicate things. However, it should at least be possible to backport ast.Keyword and ast.Slice which only added position information in 3.9. (backport to 3.8)
• Unify handling of lineno for ast.FunctionDef and ast.ClassDef nodes with decorators.
• Add custom nodes for string attributes to include positional information. Could help with the position attribute and replace token parsing -> #1393
• Changes from string attributes to dedicated notes which have been planned in CPython, but (not yet) been done. E.g. https://bugs.python.org/issue43994 -> MatchAs, MatchStar, MatchMapping.

Resources

• https://devguide.python.org/compiler/
• https://github.com/python/typed_ast
• https://github.com/python-compiler-tools/ast-compat