Report EmptyStackException as error

[zwille]

I have written a C-like language grammar using '{' '}' for nesting blocks. The lexer pushes the DEFAULT_MODE on '{' and pops it on '}' I'm using this "trick" for parsing embedded expressions in strings (island language), like so: "xxx{a+b}xx".

Now when the input is not nested well, a EmptyStackException will fire during parsing. Consider the following pseudo token stream:

{ -> pushMode DEFAULT_MODE
    ...
    { -> pushMode DEFAULT_MODE
    ...
    } -> popMode
    } -> popMode // too many '}'s
} -> popMode -> EmptyStackException

Here is the call stack:

        at org.antlr.v4.runtime.Lexer.popMode(Lexer.java:192)
	at org.antlr.v4.runtime.atn.LexerPopModeAction.execute(LexerPopModeAction.java:58)
	at org.antlr.v4.runtime.atn.LexerActionExecutor.execute(LexerActionExecutor.java:168)
	at org.antlr.v4.runtime.atn.LexerATNSimulator.accept(LexerATNSimulator.java:369)
	at org.antlr.v4.runtime.atn.LexerATNSimulator.failOrAccept(LexerATNSimulator.java:302)
	at org.antlr.v4.runtime.atn.LexerATNSimulator.execATN(LexerATNSimulator.java:233)
	at org.antlr.v4.runtime.atn.LexerATNSimulator.match(LexerATNSimulator.java:117)
	at org.antlr.v4.runtime.Lexer.nextToken(Lexer.java:141)
	at org.antlr.v4.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:169)
	at org.antlr.v4.runtime.BufferedTokenStream.fill(BufferedTokenStream.java:485)
	at org.antlr.v4.gui.TestRig.process(TestRig.java:174)
	at org.antlr.v4.gui.TestRig.process(TestRig.java:166)
	at org.antlr.v4.gui.TestRig.main(TestRig.java:119)

An error message with the token position would be helpful, instead of the exception. Maybe the lexer should recover after the exception (in DEFAULT_MODE).

[ericvergnaud]

This looks more like a parser activity than a lexer one, and using a parser you can customise error handling to your articular needs.

[KvanTTT]

I think adding an error message with correct recovery to the outermost mode (DEFAULT) is more expected and pleasant behavior.

[KvanTTT]

pushMode and popMode commands are native grammar syntax, they are not custom actions or predicates. That's why their using should not cause any runtime exceptions in any combination.

[parrt]

ok, i'll hang until you have a proposed fix.

[KitsuneAlex]

An easy workaround for this issue for the time being is adding some inline code to your lexer grammar that conditionally pops the mode stack. For Java this may look like this:

@members {
    private void popModeIfPresent() {
        if(_modeStack.size() == 0) {
            return;
        }
        popMode();
    }
}

Of course you can add more sophisticated recovery behaviour too.