javascript cpp port crashes when parsing a js file

[nj00001]

this is my js file content, and I know there is something wrong with the syntax of the following code.

"use strict";
//------------------------------------------------------------------------------
// Expression Bodies
// More expressive closure syntax.
// http://es6-features.org/#ExpressionBodies
//------------------------------------------------------------------------------

odds  = evens.map(v => v + 1)
pairs = evens.map(v => ({ even: v, odd: v + 1 }))
nums  = evens.map((v, i) => v + i)

//------------------------------------------------------------------------------
// Statement Bodies
// More expressive closure syntax.
// http://es6-features.org/#StatementBodies
//------------------------------------------------------------------------------

nums.forEach(v => {
   if (v % 5 === 0)
       fives.push(v)
})

//------------------------------------------------------------------------------
// Lexical this
// More intuitive handling of current object context.
// http://es6-features.org/#Lexicalthis
//------------------------------------------------------------------------------

this.nums.forEach((v) => {
    if (v % 5 === 0)
        this.fives.push(v)
})

this is my antlr4 code, its crashed in parser.program()

ANTLRInputStream input(targetString);
JavaScriptLexer lexer(&input);
CommonTokenStream commonTokens(&lexer);

JavaScriptParser parser(&commonTokens);

tree::ParseTree *tree = parser.program();

[kaby76]

What grammar are you using? Neither javascript/javascript nor javascript/jsx even compile for Cpp. The testing of both of these grammars is turned off.

[lizhenhua20]

What grammar are you using? Neither javascript/javascript nor javascript/jsx even compile for Cpp. The testing of both of these grammars is turned off.

this grammar https://github.com/antlr/grammars-v4/tree/master/javascript/javascript/Cpp

[kaby76]

I added a "transformGrammars.py" and the target actually builds fine. (I don't read reame.md, and the build needs it in a script that standardized across all targets). The lexer works fine, but the parser crashes, even before it gets to call any action methods on the parser base class.

It's crashing in a comparison function for arrays of PredictionContext. Geezus.

>	Test.exe!antlr4::atn::operator==(const antlr4::atn::PredictionContext & lhs, const antlr4::atn::PredictionContext & rhs) Line 206	C++	Symbols loaded.
 	Test.exe!`anonymous namespace'::predictionContextEqual(const std::shared_ptr<antlr4::atn::PredictionContext const> & lhs, const std::shared_ptr<antlr4::atn::PredictionContext const> & rhs) Line 26	C++	Symbols loaded.
 	Test.exe!std::equal<std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>>,std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>>,bool (__cdecl*)(std::shared_ptr<antlr4::atn::PredictionContext const> const &,std::shared_ptr<antlr4::atn::PredictionContext const> const &)>(const std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>> _First1, const std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>> _Last1, const std::_Vector_const_iterator<std::_Vector_val<std::_Simple_types<std::shared_ptr<antlr4::atn::PredictionContext const>>>> _First2, bool(*)(const std::shared_ptr<antlr4::atn::PredictionContext const> &, const std::shared_ptr<antlr4::atn::PredictionContext const> &) _Pred) Line 4678	C++	Symbols loaded.
 	Test.exe!antlr4::atn::ArrayPredictionContext::equals(const antlr4::atn::PredictionContext & other) Line 78	C++	Symbols loaded.
 	Test.exe!antlr4::atn::operator==(const antlr4::atn::PredictionContext & lhs, const antlr4::atn::PredictionContext & rhs) Line 207	C++	Symbols loaded.
 	Test.exe!antlr4::atn::PredictionContext::mergeArrays(std::shared_ptr<antlr4::atn::ArrayPredictionContext const> a, std::shared_ptr<antlr4::atn::ArrayPredictionContext const> b, bool rootIsWildcard, antlr4::atn::PredictionContextMergeCache * mergeCache) Line 411	C++	Symbols loaded.
 	Test.exe!antlr4::atn::PredictionContext::merge(std::shared_ptr<antlr4::atn::PredictionContext const> a, std::shared_ptr<antlr4::atn::PredictionContext const> b, bool rootIsWildcard, antlr4::atn::PredictionContextMergeCache * mergeCache) Line 210	C++	Symbols loaded.
 	Test.exe!antlr4::atn::ATNConfigSet::add(const std::shared_ptr<antlr4::atn::ATNConfig> & config, antlr4::atn::PredictionContextMergeCache * mergeCache) Line 64	C++	Symbols loaded.
 	Test.exe!antlr4::atn::ATNConfigSet::add(const std::shared_ptr<antlr4::atn::ATNConfig> & config) Line 38	C++	Symbols loaded.
 	Test.exe!antlr4::atn::PredictionModeClass::hasSLLConflictTerminatingPrediction(antlr4::atn::PredictionMode mode, antlr4::atn::ATNConfigSet * configs) Line 66	C++	Symbols loaded.
 	Test.exe!antlr4::atn::ParserATNSimulator::computeTargetState(antlr4::dfa::DFA & dfa, antlr4::dfa::DFAState * previousD, unsigned __int64 t) Line 298	C++	Symbols loaded.
 	Test.exe!antlr4::atn::ParserATNSimulator::execATN(antlr4::dfa::DFA & dfa, antlr4::dfa::DFAState * s0, antlr4::TokenStream * input, unsigned __int64 startIndex, antlr4::ParserRuleContext * outerContext) Line 179	C++	Symbols loaded.
 	Test.exe!antlr4::atn::ParserATNSimulator::adaptivePredict(antlr4::TokenStream * input, unsigned __int64 decision, antlr4::ParserRuleContext * outerContext) Line 155	C++	Symbols loaded.
 	Test.exe!JavaScriptParser::singleExpression(int precedence) Line 7321	C++	Symbols loaded.
 	Test.exe!JavaScriptParser::singleExpression(int precedence) Line 7190	C++	Symbols loaded.
 	Test.exe!JavaScriptParser::expressionSequence() Line 5811	C++	Symbols loaded.
 	Test.exe!JavaScriptParser::expressionStatement() Line 2309	C++	Symbols loaded.
 	Test.exe!JavaScriptParser::statement() Line 812	C++	Symbols loaded.
 	Test.exe!JavaScriptParser::sourceElement() Line 647	C++	Symbols loaded.
 	Test.exe!JavaScriptParser::sourceElements() Line 4908	C++	Symbols loaded.
 	Test.exe!JavaScriptParser::program() Line 604	C++	Symbols loaded.
 	Test.exe!DoParse(antlr4::CharStream * str, std::string input_name, int row_number) Line 76	C++	Symbols loaded.
 	Test.exe!ParseFilename(std::string input, int row_number) Line 116	C++	Symbols loaded.
 	Test.exe!TryParse(std::vector<std::string,std::allocator<std::string>> & args) Line 159	C++	Symbols loaded.
 	Test.exe!main(int argc, const char * * argv) Line 177	C++	Symbols loaded.
 	Test.exe!invoke_main() Line 79	C++	Symbols loaded.
 	Test.exe!__scrt_common_main_seh() Line 288	C++	Symbols loaded.
 	Test.exe!__scrt_common_main() Line 331	C++	Symbols loaded.
 	Test.exe!mainCRTStartup(void * __formal) Line 17	C++	Symbols loaded.
 	kernel32.dll!00007ffc73bd244d()	Unknown	No symbols loaded.
 	ntdll.dll!00007ffc74c4dfb8()	Unknown	No symbols loaded.

[parrt]

Is it related to this bug I added https://github.com/antlr/antlr4/issues/3845 ?

[kaby76]

Is it related to this bug I added antlr/antlr4#3845 ?

Yes, it ooks similar. The code is trying to compare two null array ptrs of predition context. I tried a couple things to fix, not as easy as I thought.

[parrt]

Yep, that sounds right. I gave up on trying to fix that bug when adding a print (cout) statement caused literally hundreds of lines of error messages to come out of clang. C++ is so stinky and the tools are so horrible.

[kaby76]

Yep, that sounds right. I gave up on trying to fix that bug when adding a print (cout) statement caused literally hundreds of lines of error messages to come out of clang. C++ is so stinky and the tools are so horrible.

I'm over in Windows, and the tools are okay--after torturing us in trying to get it right after 30 years of using us as test subjects. The problem is this nasty std::shared_ptr<>, etc., ref counting pointers. It was fine when we implemented it ourselves, or used raw pointers and used tools that were developed to fix these problems.

Anyways, I thinkk I have a solution.

[kaby76]

Try adding this before this line.

	  if (lhs == nullptr && rhs == nullptr)
		  return true;
	  if (lhs == nullptr && !(rhs == nullptr))
		  return false;
	  if (!(lhs == nullptr) && rhs == nullptr)
		  return false;

You don't want to dereference a pointer without checking that it's null first. Fixes this issue, but maybe not completely.

[kaby76]

Yeah, the regression tests for the grammar aren't all passing.

[kaby76]

The Cpp target for the javascript/javascript was incorrectly translated in the port from Java to Cpp.

This test is wrong: https://github.com/antlr/grammars-v4/blob/c4775c1c07ef227055682c65cece28add2233782/javascript/javascript/Cpp/JavaScriptLexerBase.cpp#L62

Original commit with the bogus test:

https://github.com/antlr/grammars-v4/commit/6a5d54769dbd541ef28a3a7f4b3f72108da1b64f#diff-6ba0edd9b0fa999f7902f8f2c153d4bd2b9cde2f4ee1b1314a227213426b6c1dR55

Contrast this with the Java port. https://github.com/antlr/grammars-v4/blob/c4775c1c07ef227055682c65cece28add2233782/javascript/javascript/Java/JavaScriptLexerBase.java#L98

If you want to keep the type of lastToken as a bool, then the test should be !lastToken || blah blah blah, not lastToken || blah blah blah. I don't know why the author decided to be "smart" here and mutate the type of lastToken, which is a token in every other port, to a bool.

All but one of the regression tests for javascript/javascript/examples now parses fine.

[kaby76]

There are a couple of the same errors with lastToken in the Cpp port, where lastToken == null was rewritten erroneously lastToken. Fixing these, the parser works now fine. One significant bug in the Antlr4 runtime needs to be fixed and also several in the port of the grammar.

[parrt]

if (lhs == nullptr && rhs == nullptr)

dang. nope. says can't compare lhs (nonpointer type PredictionContext) to a nullptr. Somehow those refs have become null due to array comparison in C++ lib. grrr....

oh. i put in wrong place. hang on. works!!! thanks, @kaby76