Use Python AST for rewrites

[apryor6]

For tasks involving modification of existing files, we should move to Python's abstract syntax tree module (AST).

[apryor6]

Some context:

The existing functionality for code manipulation is directly inspecting and manipulating the string contents of the source code, but this has a number of problems and is very brittle.

It's perhaps easy to see why manually manipulating source code strings is very messy. A simple goal of wanting to find the call to Flask and check for a parameter, creating one if it does not exist requires considering multi-line definitions, parameters provided by keywords, comments at any point, whether or not the invocation is inside a function or uses another function or imports another function from a different file, etc. It is extremely difficult to catch all edge cases, and I am not intending to pursue this methodology further as it is completely impractical. For that reason I consider the current implementation a prototype and have plans to migrate to a more scalable system.

One possibility is to move towards using Abstract Syntax Trees (AST). The idea here would be to identify relevant snippets of source code or entire files that are candidates for modification, parse them into an AST, and then use one of several methods from converting AST back to source code.

There would be still be plenty of complexity still even with this dramatic simplification such as comments, preservation of formatting (support for configurable formatters through which the regenerated source code could be passed?), functions in other files, global variables, etc.