Better Documentation for Python(3)

[lapplislazuli]

Hi,

I am just getting started with Antlr, and I want to try some AST Manipulation for Python. Honestly I am having a hard time, but I think everything is working I am just using it wrong.

For now I came from the getting started and wanted to parse a python3 file to get a similar tree shown.

So there are a few things when starting from the toplevel repository that I encountered:

• When I go to the Python-Folder, I see many different versions. It would be nice to get a short statement how I choose the right grammar/subfolder. Particularly the existence of Python, Python2 and Python3 is a bit confusing.
• "Just running" Antlr worked, I did that for both Lexer and Parser, but the javac Python*.java did not work as the Python3LexerBase.java was not found. I had to copy it upwards, but ... I think that is not how this should be solved.
• After this, compilations worked fine
• Using grun Python3Parser -gui sample.py fails with

Exception in thread "main" java.lang.ClassCastException: class Python3Parser
	at java.base/java.lang.Class.asSubclass(Class.java:3851)
	at org.antlr.v4.gui.TestRig.process(TestRig.java:135)
	at org.antlr.v4.gui.TestRig.main(TestRig.java:119)

I think I need some toplevel parser class too? Or I have to use it in a certain context?

• Using grun Python3Lexer -gui sample.py does fail as there is no Python3LexerParser but that seems reasonable
• From the Wiki I found This example project but it is a full maven program. Does that imply that I cannot/should not use the commandline?

To sum it up: As someone new to Antlr, coming from the getting started is a bit confusing for three reasons

1. The Getting-Started approach does not work
2. I have a hard time selecting the right grammar and I am rather guessing which one is good
3. The example from the wiki is very specific for Maven and does not give me explanations what is happening

[kaby76]

The grammars in grammars-v4/python/ are a mess. This should be cleaned up in "target-independent" format, which means this, plus organizing the target-specific code in CSharp/, Java/, Go/, Python3/, Python2/, Dart/, Cpp/, JavaScript/, etc. subdirectories. See this as an example.

The procedure should be:

mkdir foo; cd foo;
echo "Copy all base files first."
cp ........./grammars-v4/python/python3/{*.g4,*.md,*.xml,examples} .
echo "Overlay target-specific files on top of base files."
cp ........./grammars-v4/python/python3/Java/* .
echo "Run Antlr4 tool to generate Java source."
java -jar ~/Downloads/antlr-4.9.2-complete.jar -encoding utf-8  Python3Lexer.g4
java -jar ~/Downloads/antlr-4.9.2-complete.jar -encoding utf-8  Python3Parser.g4
echo "Compile Java code."
javac -cp ~/Downloads/antlr-4.9.2-complete.jar:. Python3Lexer.java
javac -cp ~/Downloads/antlr-4.9.2-complete.jar:. Python3Parser.java
echo "Run Antlr4 test rig with GUI."
java -cp ~/Downloads/antlr-4.9.2-complete.jar:. org.antlr.v4.gui.TestRig Python3 file_input -gui sample.py

The above script works--I just tested this in Msys2 on Windows11. And if you really want to be slick, just use Trash, which is used in the GitHub Actions in this repo, to generate a complete driver/parser in the targets supported by this grammar, which is only CSharp and Java. Trash can do everything that the Test Rig does, and much, much more.

There are many target-specific versions like grammars-v4/python/python3-cpp/, grammars-v4/python/python3-js/, etc. This can all be handled in one target-independent manner, with the old code stuffed into a directory called "obsolete".

[lapplislazuli]

@kaby76 Thank you very much ! I can confirm I get the GUI now (On MacOs 12). I think the biggest missing piece was the last command java -cp ~/Downloads/antlr-4.9.2-complete.jar:. org.antlr.v4.gui.TestRig Python3 file_input -gui sample.py. Can you help me to understand why the first argument is Python3 and not Python3Parser (as is the name in Python3Parser.g4)?

[kaby76]

Can you help me to understand why the first argument is Python3 and not Python3Parser (as is the name in Python3Parser.g4)?

The code that is executed, here, computes the names of the classes to load. The generated code from the tool is in Python3Parser.java/class Python3Lexer.java/class. It uses Java runtime reflection to find and load the parser/lexer code. If you pass Python3Parser, it tries to load Python3ParserParser Python3ParserLexer, which don't exist.