Better Token Stream

[x4e]

Current token stream works well, but leaves a lot to be desired.

Here are some examples of areas that could be improved:

• The reference in an import is "unclassified"
• References in new and cast operator are "unclassified"
• Semicolons are "unclassified"
• Spaces are "unclassified", when "seperator" sounds more appropriate?
• Brackets are "seperator" which seems wrong, perhaps needs a better name
• There seem to be a lot of "unclassified" tokens with just empty text

Overall I think that rather than this stream of tokens that are pretty much just wrapped text, a better approach might be to create a proper ast format.

For example a cast could be expressed as an operator class with a field representing the type of the cast, rather than the current method which is [(seperator: "("), (unclassified: "java.lang.Object"), (seperator: ")")]

Obviously this is a breaking change so I'm not proposing removing the old token stream, simply adding an option for a new sink that instead excepts an AST format.

This format could probably even be the format that you already store the final result internally, before converting it into a string.

[leibnitz27]

This format could probably even be the format that you already store the final result internally

Definitely not keen on this - I change this stuff around a lot, and don't want to have to make any guarantees that what I have is an api - that's the main reason the token stream is quite 'weak'.

I'm not averse to creating an AST, though it's quite a lot of effort. I've rather boxed myself into a corner in as much as I can't use the standard java ones, because CFR decomplies java higher than the version it's running on...

[x4e]

Ok sure, it can stay as a seperate API then.

Im happy to try and implement it in a fork over the next week or so.

[x4e]

This will take longer than expected as I'm quite busy and it's a complex task to design a good API. Still planning on completing it however.