eval('=begin') raises java.lang.IndexOutOfBoundsException

[tompng]

Found in ruby/irb CI

eval("=begin\n") #=> (eval):1: embedded document meets end of file (SyntaxError), expected
eval("=begin") #=> index is out of bounds! (java.lang.IndexOutOfBoundsException), unexpected

% ruby -v
truffleruby 23.0.0, like ruby 3.1.3, Oracle GraalVM Native [aarch64-darwin]
% ruby -e "eval('=begin')"

truffleruby: an internal exception escaped out of the interpreter,
please report it to https://github.com/oracle/truffleruby/issues.

```
index is out of bounds! (java.lang.IndexOutOfBoundsException)
	from com.oracle.truffle.api.strings.InternalErrors.indexOutOfBounds(InternalErrors.java:56)
	from com.oracle.truffle.api.strings.AbstractTruffleString.boundsCheckI(AbstractTruffleString.java:517)
	from com.oracle.truffle.api.strings.AbstractTruffleString.boundsCheckByteIndexS0(AbstractTruffleString.java:485)
	from com.oracle.truffle.api.strings.TStringInternalNodes$ReadByteNode.doRest(TStringInternalNodes.java:634)
	from com.oracle.truffle.api.strings.TStringInternalNodesFactory$ReadByteNodeGen$Uncached.execute(TStringInternalNodesFactory.java:1752)
	from com.oracle.truffle.api.strings.TruffleString$ReadByteNode.doRead(TruffleString.java:3064)
	from com.oracle.truffle.api.strings.TruffleStringFactory$ReadByteNodeGen$Uncached.execute(TruffleStringFactory.java:2488)
	from com.oracle.truffle.api.strings.AbstractTruffleString.readByteUncached(AbstractTruffleString.java:713)
	from org.truffleruby.parser.lexer.RubyLexer.p(RubyLexer.java:3233)
	from org.truffleruby.parser.lexer.RubyLexer.yylex(RubyLexer.java:921)
	from org.truffleruby.parser.lexer.RubyLexer.nextToken(RubyLexer.java:368)
	from org.truffleruby.parser.parser.RubyParser.yyparse(RubyParser.java:983)
	from org.truffleruby.parser.parser.RubyParser.yyparse(RubyParser.java:935)
	from org.truffleruby.parser.parser.RubyParser.parse(RubyParser.java:4005)
	from org.truffleruby.parser.TranslatorDriver.parseToJRubyAST(TranslatorDriver.java:393)
	from org.truffleruby.parser.TranslatorDriver.lambda$parse$0(TranslatorDriver.java:180)
	from org.truffleruby.debug.MetricsProfiler.callWithMetrics(MetricsProfiler.java:42)
	from org.truffleruby.parser.TranslatorDriver.parse(TranslatorDriver.java:177)
	from org.truffleruby.language.loader.CodeLoader.parse(CodeLoader.java:86)
	from org.truffleruby.core.kernel.KernelNodes$EvalInternalNode.parse(KernelNodes.java:810)
	from org.truffleruby.core.kernel.KernelNodesFactory$EvalInternalNodeGen.executeAndSpecialize(KernelNodesFactory.java:3377)
	from org.truffleruby.core.kernel.KernelNodesFactory$EvalInternalNodeGen.execute(KernelNodesFactory.java:3324)
	from org.truffleruby.core.kernel.KernelNodes$EvalPrepareArgsNode.eval(KernelNodes.java:731)
	from org.truffleruby.core.kernel.KernelNodesFactory$EvalPrepareArgsNodeFactory$EvalPrepareArgsNodeGen.executeAndSpecialize(KernelNodesFactory.java:3233)
	from org.truffleruby.core.kernel.KernelNodesFactory$EvalPrepareArgsNodeFactory$EvalPrepareArgsNodeGen.execute(KernelNodesFactory.java:3198)
	from org.truffleruby.language.methods.CallInternalMethodNode.alwaysInlined(CallInternalMethodNode.java:118)
	from org.truffleruby.language.methods.CallInternalMethodNodeGen.executeAndSpecialize(CallInternalMethodNodeGen.java:238)
	from org.truffleruby.language.methods.CallInternalMethodNodeGen.execute(CallInternalMethodNodeGen.java:127)
	from org.truffleruby.language.dispatch.DispatchNode.dispatchInternal(DispatchNode.java:321)
	from org.truffleruby.language.dispatch.DispatchNode.dispatch(DispatchNode.java:288)
	from org.truffleruby.language.dispatch.RubyCallNode.doCall(RubyCallNode.java:178)
	from org.truffleruby.language.dispatch.RubyCallNode.execute(RubyCallNode.java:134)
	from org.truffleruby.language.control.SequenceNode.execute(SequenceNode.java:37)
	from org.truffleruby.language.RubyMethodRootNode.execute(RubyMethodRootNode.java:65)
	from org.truffleruby.language.RubyTopLevelRootNode.execute(RubyTopLevelRootNode.java:44)
```

[eregon]

Thank you for the report. This seems a bug in the JRuby lexer, but for some reason it does not seem to repro on JRuby even though the code looks the same: https://github.com/jruby/jruby/blob/53086a956855f2fdae637ca3dbb77cb61ebd9f06/core/src/main/java/org/jruby/lexer/yacc/RubyLexer.java#L1053 and https://github.com/jruby/jruby/blob/master/core/src/main/java/org/jruby/lexer/LexingCommon.java#L543 and https://github.com/jruby/jruby/blob/master/core/src/main/java/org/jruby/util/ByteList.java#L632 Maybe by luck the byte[] is slightly bigger and then it accidentally works on JRuby.

I expect this will be solved by adopting YARP. We could change p() to deal with out-of-bounds, but not clear if returning 0 is OK, and this might have a significant perf impact on lexing.

Is this blocking you or it's OK to wait that YARP gets adopted?

[tompng]

It's not blocking, it's ok to wait YARP. I can pend the test or add "\n" to avoid it in IRB's test.

[tompng]

The exit status of ruby -e "eval('=begin')" is 1 but ruby -e "at_exit{eval('=begin')}" is 0.

$ ruby -e "eval('=begin')" 2> /dev/null; echo $?
1
$ ruby -e "at_exit{eval('=begin')}" 2> /dev/null; echo $?
0

Could it be another potential bug?

[eregon]

It is a bug, indeed, will be fixed in https://github.com/oracle/truffleruby/pull/3141

[andrykonchin]

The issue was fixed in TruffleRuby 24.0 by switching to the Prism parser.