Tsql Spaces are deleted

[qt-wu]

When using tsql parsing, spaces are eliminated in the parsing statement

My sql statement is SELECT * FROM Customers WHERE CustomerName IN ('Jane Smith');, but the parsing statement is SELECT*FROMCustomersWHERECustomerNameIN('Jane Smith');

[qt-wu]

[bkiers]

They are discarded during tokenization: https://github.com/antlr/grammars-v4/blob/master/antlr/antlr4/examples/TSqlLexer.g4#L846

If you want to keep them, change -> skip into -> channel(HIDDEN)

[KvanTTT]

If you want to keep them, change -> skip into -> channel(HIDDEN)

I don't think this is the problem. HIDDEN tokens are not included in parse tree anyway. The correct way to extract plain text over tree node is about getting startOffset and endOffset from left and right tree node's tokens and getting substring from plain text over this location.

[bkiers]

They're not in the parse tree, but the hidden tokens are being displayed when calling ParseTreeContext#ToString/GetText. At least, it used to work like that, did not try it now.

[KvanTTT]

Ok. But anyway I think -> channel(HIDDEN) should be used always and -> skip should be deprecated at all.

[bkiers]

Ok.

~Right. So it is still the case, it seems.~ (see comment below)

But anyway I think -> channel(HIDDEN) should be used always and -> skip should be deprecated at all.

That is a bit besides the point. The question was why the spaces are not shown when calling GetText(). Feel free to create a PR that replaces the -> skip in said grammar 😉

[bkiers]

EDIT: just tried the Java runtime, and ParserRuleContext#getText() does not include the text from the hidden tokens (anymore?). @qt-wu, you could try with C#, but I'd guess that does not include the text from hidden token as well.