antlr
python grammer not work
I am using Python Grammar version 1.13.2. My program is also equipped with the antl4 compiler, but I am getting many errors as shown in the picture.
I am using Python Grammar version 1.13.2. My program is also equipped with the antl4 compiler, but I am getting many errors as shown in the picture.
@RobEin see
I don't know what environment you are using the Python 3.13.2 lexer grammar in. It could be a file conflict generated by a previous version of the grammar. First of all, I suggest you delete the followings: files:
- PythonLexer.interp
- PythonParser.interp
- PythonLexer.tokens
- PythonParser.tokens
- PythonLexer.py
- PythonParser.py
- PythonParserListener.py
dirs:
- ./.antlr
- ../.antlr
And a re-generation might fix it.
Also note that there are breaking changes: dedicated tokens for soft keywords instead of NAME token:
- NAME_OR_TYPE
- NAME_OR_MATCH
- NAME_OR_CASE
- NAME_OR_WILDCARD
In case of Java target, the following files should be deleted::
-
PythonLexer.interp
-
PythonParser.interp
-
PythonLexer.tokens
-
PythonParser.tokens
-
PythonLexer.java
-
PythonParser.java
-
PythonParserBaseListener.java
-
PythonParserListener.java
-
PythonParserBaseVisitor.java
-
PythonParserVisitor.java
The picture posted is cropped so it's not clear what IDE the poster is using. But it looks like Intellij with the Antlr plug in. This is because there are red squiggles under the grammar symbols. This happens when you do not "generate" the parser through the UI.
The picture posted is cropped so it's not clear what IDE the poster is using. But it looks like Intellij with the Antlr plug in. This is because there are red squiggles under the grammar symbols. This happens when you do not "generate" the parser through the UI. @kaby76 im using https://github.com/HanzoDev1375/Ghostide
In case of Java target, the following files should be deleted::
- PythonLexer.interp
- PythonParser.interp
- PythonLexer.tokens
- PythonParser.tokens
- PythonLexer.java
- PythonParser.java
- PythonParserBaseListener.java
- PythonParserListener.java
- PythonParserBaseVisitor.java
- PythonParserVisitor.java
I only use the lexer parser. It worked fine in previous versions, but not in the new version. Please note that the new code has not been uploaded yet. It will be uploaded in the next few days.
https://github.com/HanzoDev1375/Ghostide/blob/main/app/src/main/java/io/github/rosemoe/sora/langs/python/PythonCodeAnalyzer.java
For an example
All we see is that there are red squiggles denoting errors in an IDE of the .g4 file, not of a Python3 input file. In other words we don't see the python3 lexer being generated, compiled, or run on a python3 input file. The python grammar was tested in Github Actions and it passed without error. https://github.com/antlr/grammars-v4/actions/runs/13247965899/job/36978984495?pr=4410
All we see is that there are red squiggles denoting errors in an IDE of the .g4 file, not of a Python3 input file. In other words we don't see the python3 lexer being generated, compiled, or run on a python3 input file. The python grammar was tested in Github Actions and it passed without error. https://github.com/antlr/grammars-v4/actions/runs/13247965899/job/36978984495?pr=4410
Can you test in Java?
Here's a PR with the build and test of python3_13. https://github.com/antlr/grammars-v4/actions/runs/13388160812/job/37389572122?pr=4417. I even tested it locally on my machine.
02/18-05:28:06 ~/issues/g4-4416/python/python3_13
$ dotnet tool restore
Tool 'trcaret' (version '0.23.11') was restored. Available commands: trcaret
Tool 'trcover' (version '0.23.11') was restored. Available commands: trcover
Tool 'trgen' (version '0.23.11') was restored. Available commands: trgen
Tool 'trglob' (version '0.23.11') was restored. Available commands: trglob
Tool 'triconv' (version '0.23.11') was restored. Available commands: triconv
Tool 'trparse' (version '0.23.11') was restored. Available commands: trparse
Tool 'trquery' (version '0.23.11') was restored. Available commands: trquery
Tool 'trtext' (version '0.23.11') was restored. Available commands: trtext
Tool 'trwdog' (version '0.23.11') was restored. Available commands: trwdog
Tool 'trxgrep' (version '0.23.11') was restored. Available commands: trxgrep
Tool 'trxml' (version '0.23.11') was restored. Available commands: trxml
Tool 'trxml2' (version '0.23.11') was restored. Available commands: trxml2
Tool 'trclonereplace' (version '0.23.11') was restored. Available commands: trclonereplace
Tool 'trcombine' (version '0.23.11') was restored. Available commands: trcombine
Tool 'trconvert' (version '0.23.11') was restored. Available commands: trconvert
Tool 'trfoldlit' (version '0.23.11') was restored. Available commands: trfoldlit
Tool 'trgenvsc' (version '0.23.11') was restored. Available commands: trgenvsc
Tool 'tritext' (version '0.23.11') was restored. Available commands: tritext
Tool 'trjson' (version '0.23.11') was restored. Available commands: trjson
Tool 'trperf' (version '0.23.11') was restored. Available commands: trperf
Tool 'trrename' (version '0.23.11') was restored. Available commands: trrename
Tool 'trsort' (version '0.23.11') was restored. Available commands: trsort
Tool 'trsplit' (version '0.23.11') was restored. Available commands: trsplit
Tool 'trsponge' (version '0.23.11') was restored. Available commands: trsponge
Tool 'trtokens' (version '0.23.11') was restored. Available commands: trtokens
Tool 'trtree' (version '0.23.11') was restored. Available commands: trtree
Tool 'trunfold' (version '0.23.11') was restored. Available commands: trunfold
Restore was successful.
02/18-05:28:12 ~/issues/g4-4416/python/python3_13
$ dotnet trgen -t Java
C:\msys64\home\Kenne\issues\g4-4416\python\python3_13
CSharp PythonParser.g4 success 0.0486789
CSharp PythonLexer.g4 success 0.0296834
Rendering template file from Java/st.build.ps1 to ./Generated-Java-0/st.build.ps1
Rendering template file from Java/st.build.sh to ./Generated-Java-0/st.build.sh
Rendering template file from Java/st.clean.ps1 to ./Generated-Java-0/st.clean.ps1
Rendering template file from Java/st.clean.sh to ./Generated-Java-0/st.clean.sh
Rendering template file from Java/st.ErrorListener.java to ./Generated-Java-0/st.ErrorListener.java
Rendering template file from Java/st.makefile to ./Generated-Java-0/st.makefile
Rendering template file from Java/st.MyDiagnosticErrorListener.java to ./Generated-Java-0/st.MyDiagnosticErrorListener.java
Rendering template file from Java/st.perf.sh to ./Generated-Java-0/st.perf.sh
Rendering template file from Java/st.run.ps1 to ./Generated-Java-0/st.run.ps1
Rendering template file from Java/st.run.sh to ./Generated-Java-0/st.run.sh
Rendering template file from Java/st.Test.java to ./Generated-Java-0/st.Test.java
Rendering template file from Java/st.test.ps1 to ./Generated-Java-0/st.test.ps1
Rendering template file from Java/st.test.sh to ./Generated-Java-0/st.test.sh
Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/README.md to ./Generated-Java-0/README.md
Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/PythonParser.g4 to ./Generated-Java-0/PythonParser.g4
Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/PythonLexer.g4 to ./Generated-Java-0/PythonLexer.g4
Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/Python3_13_2_official_grammar.peg to ./Generated-Java-0/Python3_13_2_official_grammar.peg
Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/pom.xml to ./Generated-Java-0/pom.xml
Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/desc.xml to ./Generated-Java-0/desc.xml
Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/changes.md to ./Generated-Java-0/changes.md
Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/Java/PythonLexerBase.java to ./Generated-Java-0/PythonLexerBase.java
02/18-05:28:34 ~/issues/g4-4416/python/python3_13
$ cd Generated-Java-0/
02/18-05:28:37 ~/issues/g4-4416/python/python3_13/Generated-Java-0
$ make
bash build.sh
02/18-05:28:46 ~/issues/g4-4416/python/python3_13/Generated-Java-0
$ bash run.sh ../examples/*.py
Java 0 ../examples/__future__.py success 0.101
Java 1 ../examples/__hello__.py success 0.004
Java 2 ../examples/_aix_support.py success 0.071
Java 3 ../examples/_android_support.py success 0.037
Java 4 ../examples/_collections_abc.py success 0.135
Java 5 ../examples/_colorize.py success 0.006
Java 6 ../examples/_compat_pickle.py success 0.032
Java 7 ../examples/_compression.py success 0.009
Java 8 ../examples/_ios_support.py success 0.004
Java 9 ../examples/_markupbase.py success 0.034
Java 10 ../examples/_opcode_metadata.py success 0.006
Java 11 ../examples/_osx_support.py success 0.045
Java 12 ../examples/_py_abc.py success 0.007
Java 13 ../examples/_pydatetime.py success 0.164
Java 14 ../examples/_pydecimal.py success 0.172
Java 15 ../examples/_pyio.py success 0.147
Java 16 ../examples/_pylong.py success 0.013
Java 17 ../examples/_sitebuiltins.py success 0.009
Java 18 ../examples/_strptime.py success 0.037
Java 19 ../examples/_threading_local.py success 0.003
Java 20 ../examples/_weakrefset.py success 0.002
Total Time: 2.766
02/18-05:29:12 ~/issues/g4-4416/python/python3_13/Generated-Java-0
$
Again, the picture you give is for an IDE that shows errors while trying to parse PythonLexer.g4, not the output of trying to compile and link a parser generated in Java, or the output of trying to parse or lex a Python3 input string or file. Your IDE is having a problem; the python3_13 grammar is fine.
در اینجا یک روابط عمومی با ساخت و آزمایش python3_13 است. https://github.com/antlr/grammars-v4/actions/runs/13388160812/job/37389572122?pr=4417 . من حتی آن را به صورت محلی روی دستگاهم تست کردم.
02/18-05:28:06 ~/issues/g4-4416/python/python3_13 $ dotnet tool restore Tool 'trcaret' (version '0.23.11') was restored. Available commands: trcaret Tool 'trcover' (version '0.23.11') was restored. Available commands: trcover Tool 'trgen' (version '0.23.11') was restored. Available commands: trgen Tool 'trglob' (version '0.23.11') was restored. Available commands: trglob Tool 'triconv' (version '0.23.11') was restored. Available commands: triconv Tool 'trparse' (version '0.23.11') was restored. Available commands: trparse Tool 'trquery' (version '0.23.11') was restored. Available commands: trquery Tool 'trtext' (version '0.23.11') was restored. Available commands: trtext Tool 'trwdog' (version '0.23.11') was restored. Available commands: trwdog Tool 'trxgrep' (version '0.23.11') was restored. Available commands: trxgrep Tool 'trxml' (version '0.23.11') was restored. Available commands: trxml Tool 'trxml2' (version '0.23.11') was restored. Available commands: trxml2 Tool 'trclonereplace' (version '0.23.11') was restored. Available commands: trclonereplace Tool 'trcombine' (version '0.23.11') was restored. Available commands: trcombine Tool 'trconvert' (version '0.23.11') was restored. Available commands: trconvert Tool 'trfoldlit' (version '0.23.11') was restored. Available commands: trfoldlit Tool 'trgenvsc' (version '0.23.11') was restored. Available commands: trgenvsc Tool 'tritext' (version '0.23.11') was restored. Available commands: tritext Tool 'trjson' (version '0.23.11') was restored. Available commands: trjson Tool 'trperf' (version '0.23.11') was restored. Available commands: trperf Tool 'trrename' (version '0.23.11') was restored. Available commands: trrename Tool 'trsort' (version '0.23.11') was restored. Available commands: trsort Tool 'trsplit' (version '0.23.11') was restored. Available commands: trsplit Tool 'trsponge' (version '0.23.11') was restored. Available commands: trsponge Tool 'trtokens' (version '0.23.11') was restored. Available commands: trtokens Tool 'trtree' (version '0.23.11') was restored. Available commands: trtree Tool 'trunfold' (version '0.23.11') was restored. Available commands: trunfold Restore was successful. 02/18-05:28:12 ~/issues/g4-4416/python/python3_13 $ dotnet trgen -t Java C:\msys64\home\Kenne\issues\g4-4416\python\python3_13 CSharp PythonParser.g4 success 0.0486789 CSharp PythonLexer.g4 success 0.0296834 Rendering template file from Java/st.build.ps1 to ./Generated-Java-0/st.build.ps1 Rendering template file from Java/st.build.sh to ./Generated-Java-0/st.build.sh Rendering template file from Java/st.clean.ps1 to ./Generated-Java-0/st.clean.ps1 Rendering template file from Java/st.clean.sh to ./Generated-Java-0/st.clean.sh Rendering template file from Java/st.ErrorListener.java to ./Generated-Java-0/st.ErrorListener.java Rendering template file from Java/st.makefile to ./Generated-Java-0/st.makefile Rendering template file from Java/st.MyDiagnosticErrorListener.java to ./Generated-Java-0/st.MyDiagnosticErrorListener.java Rendering template file from Java/st.perf.sh to ./Generated-Java-0/st.perf.sh Rendering template file from Java/st.run.ps1 to ./Generated-Java-0/st.run.ps1 Rendering template file from Java/st.run.sh to ./Generated-Java-0/st.run.sh Rendering template file from Java/st.Test.java to ./Generated-Java-0/st.Test.java Rendering template file from Java/st.test.ps1 to ./Generated-Java-0/st.test.ps1 Rendering template file from Java/st.test.sh to ./Generated-Java-0/st.test.sh Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/README.md to ./Generated-Java-0/README.md Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/PythonParser.g4 to ./Generated-Java-0/PythonParser.g4 Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/PythonLexer.g4 to ./Generated-Java-0/PythonLexer.g4 Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/Python3_13_2_official_grammar.peg to ./Generated-Java-0/Python3_13_2_official_grammar.peg Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/pom.xml to ./Generated-Java-0/pom.xml Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/desc.xml to ./Generated-Java-0/desc.xml Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/changes.md to ./Generated-Java-0/changes.md Rendering template file from C:/msys64/home/Kenne/issues/g4-4416/python/python3_13/Java/PythonLexerBase.java to ./Generated-Java-0/PythonLexerBase.java 02/18-05:28:34 ~/issues/g4-4416/python/python3_13 $ cd Generated-Java-0/ 02/18-05:28:37 ~/issues/g4-4416/python/python3_13/Generated-Java-0 $ make bash build.sh 02/18-05:28:46 ~/issues/g4-4416/python/python3_13/Generated-Java-0 $ bash run.sh ../examples/*.py Java 0 ../examples/__future__.py success 0.101 Java 1 ../examples/__hello__.py success 0.004 Java 2 ../examples/_aix_support.py success 0.071 Java 3 ../examples/_android_support.py success 0.037 Java 4 ../examples/_collections_abc.py success 0.135 Java 5 ../examples/_colorize.py success 0.006 Java 6 ../examples/_compat_pickle.py success 0.032 Java 7 ../examples/_compression.py success 0.009 Java 8 ../examples/_ios_support.py success 0.004 Java 9 ../examples/_markupbase.py success 0.034 Java 10 ../examples/_opcode_metadata.py success 0.006 Java 11 ../examples/_osx_support.py success 0.045 Java 12 ../examples/_py_abc.py success 0.007 Java 13 ../examples/_pydatetime.py success 0.164 Java 14 ../examples/_pydecimal.py success 0.172 Java 15 ../examples/_pyio.py success 0.147 Java 16 ../examples/_pylong.py success 0.013 Java 17 ../examples/_sitebuiltins.py success 0.009 Java 18 ../examples/_strptime.py success 0.037 Java 19 ../examples/_threading_local.py success 0.003 Java 20 ../examples/_weakrefset.py success 0.002 Total Time: 2.766 02/18-05:29:12 ~/issues/g4-4416/python/python3_13/Generated-Java-0 $مجدداً، تصویری که ارائه میدهید مربوط به یک IDE است که خطاها را هنگام تلاش برای تجزیه PythonLexer.g4 نشان میدهد ، نه خروجی تلاش برای کامپایل و پیوند یک تجزیهکننده تولید شده در جاوا، یا خروجی تلاش برای تجزیه یا لکس یک رشته یا فایل ورودی Python3. IDE شما مشکل دارد. گرامر python3_13 خوب است.
It creates the lexer and parser files well. I get an error when I enter the code editor and the reserved keywords do not display any color. I don't know if the problem is with the lexer or my code. If you look at the rest of my program languages, it works fine.
I don't know if the problem is with the lexer or my code.
You're going to have to debug your environment, your IDE. This has nothing to do with the python3_13 grammar. The python3_13 grammar (the .g4's) parse fine. The IDE isn't parsing a valid .g4.
I tested this with official Antlr 4.13.2, 4.13.1, Trash trgen, and antlr-ng tool. All parse the .g4's fine.
All Antlr4 grammars in testing in this repo are parsed using two different parsers: (1) the official Antlr4 tool; (2) the Trash Toolkit with trgen, which uses a copy of the grammar. https://github.com/antlr/grammars-v4/tree/63d037bbb084b2a5692ef09957c5bf7e647eee97/antlr/antlr4. I am planning to add the new antlr-ng tool to this repo, and it also works fine.
I also tested the generated parser for Python3 input. Everything works fine, producing parse trees for valid Python3 input.
I get an error when I enter the code editor and the reserved keywords do not display any color.
Which reserved keywords?? The "reserved keywords" in the PythonLexer.g4 or the reserved keywords in a python3 input file?
PythonLexer.g4 is not parsed using the python3_13 grammar. I suggest you write a driver in Java that parses a python3 file, and bypass trying to test the python3_13 grammar via your IDE. Your IDE is not working; the grammar is fine.
I see that the yor IDE starts to fail at the STRING_PREFIX fragment rule.
Here I'm using a new option that I didn't use in the previous lexer grammar:
options { caseInsensitive=true; }
This might be the problem in your environment.
Try replacing the following three fragment rules with the old version without options:
fragment STRING_PREFIX options { caseInsensitive=true; } : 'r' | 'u';
fragment STRING_PREFIX : 'r' | 'u' | 'R' | 'U';
fragment BYTES_PREFIX options { caseInsensitive=true; } : 'b' | 'br' | 'rb';
fragment BYTES_PREFIX : 'b' | 'B' | 'br' | 'Br' | 'bR' | 'BR' | 'rb' | 'rB' | 'Rb' | 'RB';
fragment FSTRING_PREFIX options { caseInsensitive=true; } : 'f' | 'fr' | 'rf';
fragment FSTRING_PREFIX : 'f' | 'F' | 'fr' | 'Fr' | 'fR' | 'FR' | 'rf' | 'rF' | 'Rf' | 'RF';
The problem is that the version of the Antlr grammar for Antlr4 used to generate the Antlr4 parser in the repo at Ghostide/app/src/main/java/org/antlr/parser/antlr4/ is an old version. There is no ANTLRv4Parser.g4 checked in the repo, and the generated file ANTLRv4Parser.java was reformatted, which makes comparison to what version of the .g4 it came from difficult to determine.
Not checking in the .g4's, checking in generated files, and reformatting generated files, are each bad software development practices. This is why your IDE cannot parse the new Python3 grammar.
مشکل این است که نسخه گرامر Antlr برای Antlr4 که برای تولید تجزیه کننده Antlr4 در مخزن Ghostide/app/src/main/java/org/antlr/parser/antlr4/ استفاده میشود ، نسخه قدیمی است. هیچ ANTLRv4Parser.g4 در مخزن بررسی نشده است، و فایل ایجاد شده ANTLRv4Parser.java دوباره قالب بندی شده است، که تشخیص آن را با نسخه ای از .g4 دشوار می کند.
بررسی نکردن در .g4، بررسی فایلهای تولید شده، و قالببندی مجدد فایلهای تولید شده، هر کدام از اقدامات بد توسعه نرمافزار هستند. به همین دلیل است که IDE شما نمی تواند گرامر جدید Python3 را تجزیه کند.
Okay, I'll update my antlr4 grammar tonight to fix this error.
Excellent. Definitely check in a copy of the latest .g4's with those Java/*.java files. You might want to add a readme.md saying what git commit you are using so you know exactly what version you are using in addition to having a copy of the files. Some people do check in the .java generated so they can debug a tagged version without doing a full rebuild.
عالی حتماً یک کپی از آخرین .g4 را با آن فایل های Java/*.java بررسی کنید. ممکن است بخواهید یک readme.md اضافه کنید که بگوید از چه git commit استفاده میکنید تا علاوه بر داشتن یک کپی از فایلها، دقیقاً بدانید از چه نسخهای استفاده میکنید. برخی از افراد .java تولید شده را بررسی می کنند تا بتوانند نسخه برچسب گذاری شده را بدون انجام بازسازی کامل اشکال زدایی کنند.
https://github.com/HanzoDev1375/Ghostide/commit/9b463ad31975ed8fb575d580f6355510259d0391
I updated the grammar and it worked, but unfortunately Python lexer didn't help me with color coding. I had to create a separate grammar for my IDE because my goal is color coding. Parsing is the responsibility of another part of my program.