antlr
Control-click in ANTLR Preview gives `NoSuchElementException`
To reproduce this bug, quit IntelliJ and start up fresh. The bug is triggered only once, so the fresh startup is needed to see it the first time.
Using this grammar:
grammar Expr;
prog: expr EOF ;
expr: expr ('*'|'/') expr
| expr ('+'|'-') expr
| INT
| '(' expr ')'
;
NEWLINE : [ \r\n]+ -> skip;
INT : [0-9]+ ;
Open the ANTLR Preview (i.e. right click prog in the source, and choose "Test rule prog")
Enter the following input:
1 + 3 * 3 - 5
Hold the CTRL key, and hover over the + character, and you should see the grammar rule tool tip appear saying #1 Type '+', Line 1:2:
While still holding the CTRL key in this spot, click the mouse button. An exception occurs with this stacktrace:
java.util.NoSuchElementException: token index 85 out of range 0..75
at org.antlr.runtime.BufferedTokenStream.get(BufferedTokenStream.java:154)
at org.antlr.intellij.plugin.preview.InputPanel.setCursorToGrammarElement(InputPanel.java:562)
at org.antlr.intellij.plugin.preview.PreviewEditorMouseListener.mouseClicked(PreviewEditorMouseListener.java:46)
at com.intellij.openapi.editor.impl.EditorImpl$MyMouseAdapter.runMouseClickedCommand(EditorImpl.java:3932)
at com.intellij.openapi.editor.impl.EditorImpl$MyMouseAdapter.mouseReleased(EditorImpl.java:3850)
at java.desktop/java.awt.Component.processMouseEvent(Component.java:6656)
at java.desktop/javax.swing.JComponent.processMouseEvent(JComponent.java:3385)
at java.desktop/java.awt.Component.processEvent(Component.java:6421)
at java.desktop/java.awt.Container.processEvent(Container.java:2266)
at java.desktop/java.awt.Component.dispatchEventImpl(Component.java:5026)
at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2324)
at java.desktop/java.awt.Component.dispatchEvent(Component.java:4854)
at java.desktop/java.awt.LightweightDispatcher.retargetMouseEvent(Container.java:4948)
at java.desktop/java.awt.LightweightDispatcher.processMouseEvent(Container.java:4575)
at java.desktop/java.awt.LightweightDispatcher.dispatchEvent(Container.java:4516)
at java.desktop/java.awt.Container.dispatchEventImpl(Container.java:2310)
at java.desktop/java.awt.Window.dispatchEventImpl(Window.java:2804)
at java.desktop/java.awt.Component.dispatchEvent(Component.java:4854)
at java.desktop/java.awt.EventQueue.dispatchEventImpl(EventQueue.java:790)
at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:739)
at java.desktop/java.awt.EventQueue$3.run(EventQueue.java:731)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:97)
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:763)
at java.desktop/java.awt.EventQueue$4.run(EventQueue.java:761)
at java.base/java.security.AccessController.doPrivileged(AccessController.java:399)
at java.base/java.security.ProtectionDomain$JavaSecurityAccessImpl.doIntersectionPrivilege(ProtectionDomain.java:86)
at java.desktop/java.awt.EventQueue.dispatchEvent(EventQueue.java:760)
at com.intellij.ide.IdeEventQueue.defaultDispatchEvent(IdeEventQueue.kt:667)
at com.intellij.ide.IdeEventQueue.dispatchMouseEvent(IdeEventQueue.kt:615)
at com.intellij.ide.IdeEventQueue._dispatchEvent(IdeEventQueue.kt:570)
at com.intellij.ide.IdeEventQueue.access$_dispatchEvent(IdeEventQueue.kt:68)
at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:349)
at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1$1.compute(IdeEventQueue.kt:348)
at com.intellij.openapi.progress.impl.CoreProgressManager.computePrioritized(CoreProgressManager.java:787)
at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:348)
at com.intellij.ide.IdeEventQueue$dispatchEvent$processEventRunnable$1$1.invoke(IdeEventQueue.kt:343)
at com.intellij.ide.IdeEventQueueKt.performActivity$lambda$1(IdeEventQueue.kt:995)
at com.intellij.openapi.application.TransactionGuardImpl.performActivity(TransactionGuardImpl.java:113)
at com.intellij.ide.IdeEventQueueKt.performActivity(IdeEventQueue.kt:995)
at com.intellij.ide.IdeEventQueue.dispatchEvent$lambda$4(IdeEventQueue.kt:343)
at com.intellij.openapi.application.impl.ApplicationImpl.runIntendedWriteActionOnCurrentThread(ApplicationImpl.java:831)
at com.intellij.ide.IdeEventQueue.dispatchEvent(IdeEventQueue.kt:385)
at java.desktop/java.awt.EventDispatchThread.pumpOneEventForFilters(EventDispatchThread.java:207)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForFilter(EventDispatchThread.java:128)
at java.desktop/java.awt.EventDispatchThread.pumpEventsForHierarchy(EventDispatchThread.java:117)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:113)
at java.desktop/java.awt.EventDispatchThread.pumpEvents(EventDispatchThread.java:105)
at java.desktop/java.awt.EventDispatchThread.run(EventDispatchThread.java:92)
I'm using the latest IntelliJ:
IntelliJ IDEA 2023.1.3 (Ultimate Edition)
Build #IU-231.9161.38, built on June 20, 2023
Licensed to Broc Seib
Subscription is active until January 11, 2024.
Runtime version: 17.0.7+10-b829.16 aarch64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
macOS 13.4
GC: G1 Young Generation, G1 Old Generation
Memory: 2000M
Cores: 10
Metal Rendering is ON
Registry:
debugger.new.tool.window.layout=true
ide.experimental.ui=true
eslint.additional.file.extensions=svelte
ide.images.show.chessboard=true
Non-Bundled Plugins:
org.antlr.intellij.plugin (1.20)
org.rust.lang (0.4.197.5401-231)
com.intellij.plugins.macoskeymap (231.7515.9)
org.jetbrains.plugins.go (231.9161.14)
com.intellij.nativeDebug (231.8770.15)
String Manipulation (9.9.0)
com.dherre3.wasm-text-parser (1.0.1)
org.jetbrains.kotlin (231-1.9.0-release-358-IJ8770.65)
com.github.bjansen.intellij.pebble (0.9.1)
Kotlin: 231-1.9.0-release-358-IJ8770.65
@parrt I need help on this one. The code is basically this:
Token tokenUnderCursor = ParsingUtils.getTokenUnderCursor(previewState, offset);
Integer atnState = parser.inputTokenToStateMap.get(tokenUnderCursor);
Interval region = previewState.g.getStateToGrammarRegion(atnState);
CommonToken token = (CommonToken) previewState.g.tokenStream.get(region.a);
For some reason, atnState refers to a state that is not in the grammar, but apparently in a internal (generated?) grammar that looks like this:
expr
: ( {} INT<tokenIndex=41>
| '('<tokenIndex=45> expr<tokenIndex=47> ')'<tokenIndex=49>
)
(
{precpred(_ctx, 4)}?<p=4> ('*'<tokenIndex=20>|'/'<tokenIndex=22>) expr<tokenIndex=25,p=5>
| {precpred(_ctx, 3)}?<p=3> ('+'<tokenIndex=32>|'-'<tokenIndex=34>) expr<tokenIndex=37,p=4>
)*
;
Then region refers to a location in this grammar instead of the grammar being tested, that's what causes the exception.
I have no idea how to fix this.
wow. that's a tricky one and I don't have a huge amount of time to dig into this. I wonder if the grammar and text window are somehow out of sync or grammar is delayed getting updated. I think I saw that at some point.
Ah. yeah, most likely it has something to do with conversion of left recursive grammars. Anytime there is a left recursive grammar we convert it to a new version of the rules which is where the atnState will be referring to. going backwards to the location in the original grammar would be tricky but on guessing I have a mechanism to do that mapping if only to produce error messages from the command line. I guess we will need a check in the plugin looking for left recursive rules and, if so, look in the original grammar for the position not inthe generated grammar. I would start by poking around and seeing how I generate error messages from the command line for left recursive rules.
