JuliaLang
Remove the `startpos` field from tokens
This makes "untokenization" slightly more annoying (since you need to look at the previous token to know where to start) but I think it is worth it. Some tests needed to update based on that. It is a bit unclear if this is worth the effort.
By the way I have a separate token type in JuliaSyntax.SyntaxToken (with some fields needed by the parser) and Tokenize.Token is ephemeral when used inside ParseStream. So there shouldn't be memory overhead from having Token with a few extra fields. At some point we can look at whether to merge those types together.
I'll drop the last commit for now and put that in a separate PR since it does make untokenization a bit more awkward and perhaps that is not worth it.
I quite like what you did with the tests, overall.
The issue as you say is worse line info. Even putting the @test outside like you suggest we don't really know which one of all the checks fails.
we don't really know which one of all the checks fails
Yeah this is kinda annoying. It's why I went with the verbose style originally.
It's really a problem with @test ... it's like we need a @test_table macro or something which knows about tables of test cases (ie, the reference arrays you have here) and can report which entry in the table was the problem.
@test_tablemacro or something which knows about tables of test cases
Although... in current Julia the table would have to be a begin... end block so the macro can know which line each test is on, because a multi-line array literal doesn't come with line number nodes :-/
maybe use https://github.com/JuliaLang/julia/pull/46138
well yeah if stdlib versioning wasn't tied to Julia it might be useful :-/