Provide access to hidden tokens in a stream

[sergun]

Hi!

First of all thanks a lot for this great tool!

My question is: I need (in Python) CommonTokenStream(lexer) class instance to get access to hidden tokens like comments with .getHiddenTokensToLeft/Right(...) methods of that stream. Am I right there is no easy way / workaround do get such instance from do_parse?

Sergey.

[amykyta3]

Currently the tool only supports transcribing tokens part of the visible grammar. I purposely have it discard everything else since a lot of the processing time is actually spent translating the parsed tree & tokens into Python objects. I can look into what it would take to also transcribe these hidden tokens. I admit i have not used that feature before.

[sergun]

It would be nice if you can implement this.

BTW Do you have any ideas why ANTLR community still didn't implement cython backend? :-) Python backend is unusable in most cases.. And only projects like your makes it working. E.g. I deal with tens of SQL-scripts each with hundred of lines. Parsing of them in Python-backend takes minutes (((