Help or Feature Request: Directly Providing Tokens instead of (Re-)Parsing Text

[fmagin]

First of, thanks for the great project, it has been very helpful to me on multiple occasions already!

I'm currently writing a viewer for a certain Domain Specific Language inside Java Swing based tool. The underlying language exists as Java objects that represent a three-address code (TAC) form of an Intermediate Language (IL) used for Program Analysis. There exists no official good viewer for it.

Currently, I'm rendering each statement in this language to a line of text based on some config options, and setting these lines as the text on the RSyntaxTextArea component.

For future features like syntax highlighting and UI actions based on the precise part of the TAC (e.g. if it's the input part, the operation, or the output part) I need to tokenize it somehow. But because I'm already taking an IL that is structured, and flattening it to text, it seems unnecessarily convoluted to now write a tokenizer for these strings. Instead I want to hook into whatever mechanism that allows me to directly supply Tokens.

Is there an existing way to do this already? If not, would it be possible to implement this and what would it take to achieve this? I'm willing to build a PR for this, if this is something that's technically possible, but simple hasn't been formalized into a public interface yet.

[bobbylight]

If I'm understanding you right, you're asking if a tokenizer can look at a structure other than the text? I suppose it could, but it would have to fetch it itself, and then translate it to the text rendered in RSTA, right?

TokenMakers are really meant to be a lexer, and parse text, not other structures like ASTs, etc.

[fmagin]

I think then it would be conceptually more appropriate to provide the tokens directly to whatever code would otherwise use a TokenMaker? But overall my problem is roughly "Given an AST, render it in a nice Java Swing window", and RSyntaxArea so far has been a very good start for this, and I'm wondering how far it can go.