Implement ANTLR based TokenMaker

[tisoft]

trafficstars

Fixes #221

[coveralls]

Coverage remained the same at 70.747% when pulling d22c9918c4779c4c2ee56b8dfbcb624e484a12a0 on tisoft:antlr into 685a7eee3383c153946df46f93570068e14fb8f2 on bobbylight:master.

[tisoft]

Hi @bobbylight,

What do you think about this? Is this something you would think of as usefull?

My design goal for this was not to recreate the same token output as the existing TokenMakers have, but to use unchanged Antlr grammars as input. Most Antlr Grammars should be supported, except ones that use lexer modes, since I currently have no way to keep track of the mode stack. Multiline tokens (like Comments or Strings) should be supported, if they are correctly given in the constructor. Some grammars need a bit of extra love, like Python, since they insert dummy tokens, that need to be filtered out. And most grammars need a quick fix to support skipped tokens. All of this can be done by overriding the compiled Lexer classes. No access to the grammar files is needed. So you could still build a Syntax Highlighter for a language where you got the Lexer from a vendor as compiled class.

Next step is creating a bit of documentation on how to use it. I have added some example grammars to the demo application for a quick start.

I see the following possibilities, going forward:

1. Add it as subproject to this project
   • users would depend on "com.fifesoft:rsyntaxtextarea-antlr4:${version}", which would pull in the antlr4 runtime
   • this PR would lead to something like this
2. Merge it into the main code base
   • the antlr-runtime would need to be made optional in the pom
     • it is needed for compiling
     • it is only needed at runtime, if the user uses the antlr support
   • users would depend on "com.fifesoft:rsyntaxtextarea:${version}" and explicitly on "org.antlr:antlr4-runtime:${version}"
3. Create a new project in your namespace
   • maven coordinates would probably be the same as in 1.
4. Create a new project in my namespace
   • maven coordinates would be in the "de.tisoft" namespace

[ice1000]

Can we have the com.fifesoft:rsyntaxtextarea-antlr4:${version} thing as a third-party library?

[tisoft]

@bobbylight Would you consider adding this? Otherwise I will create an own project for it.

[tisoft]

As it seems, this is not going to get merged, I have created my own project: https://github.com/tisoft/rsyntaxtextarea-antlr4-extension

It is the same code as in this PR, just moved to my namespace. I have not yet added the examples, but the code is usable. Also documentation is missing, yet.

I'm open for feature requests, bug reports and improvements.

First release is already in Maven Central: https://search.maven.org/artifact/de.tisoft.rsyntaxtextarea/rsyntaxtextarea-antlr4-extension

I will leave this PR open, if there is interest from @bobbylight to merge this in, I'm more than happy to do so.

[bobbylight]

@tisoft Thanks for taking the initiative here, I've been stretched too thin and have neglected RSTA. This is something that's been requested for a while now, as you're aware. I hope to check out this PR soon, though it may be some time in April.

BTW, I saw you made an Antlr 6502 parser - is that out of interest on your part, or did you just grep projects for Antlr implementations for a breadth of test cases? Learning NES Assembly programming is also on my todo list :)

[tisoft]

@bobbylight Thanks for the status update!

I'll update this PR with the improvements I made and will keep it current.

I just took some example grammars, to have something for the demo program and for testing. My goal is to match every from your supported languages with an ANTLR based and have some more. 😆

[tisoft]

I have rebased against master and also updated the code with the latest changes I did in my fork.