protocompile
protocompile copied to clipboard
bufbuild
Implement comment tracking in the new lexer
This change does two things:
-
It adds support for attacking comments to tokens in the token package, as well as a way to query the actual text of a comment token. Comment tokens which are token trees are used to represent tokens that are grouped together.
-
It adds support in the lexer for automatically attaching comments. I have included a test consisting of all of the funny cases listed on protobuf.com.
As discussed elsewhere, W're going to punt this for now. I'm going to convert this to draft for future reference, but what we'll probably do is make comment attribution lazy.