antlr4
antlr4 copied to clipboard
antlr
A new lexer command: Less
Closes #212
I mostly just duplicated the code for the more command and updated the behavior as necessary. I also added an entirely new token type instead of overloading MORE.
I chose to handle rewinding the lexer in nextToken instead of directly in the less method because I had concerns about potential side effects with lexer predicates and actions.
An alternative I considered was adding a 'retry' field and having the less method do _retry = true; _type = MORE.
Also, when it comes to the command name: although the symmetry with more is nice, another thought I had was to name the command retry, which I think is more descriptive of what the command actually does.
Let me know if you have any concerns or would like me to make changes. Thanks.
@sharwell Is this an addition you would still be interested in?
FWIW, the checks did pass for the last meaningful commit (e91e700). It's just after I pulled in the updated contributors file (60cdbfe) that the checks failed.
@mike-lischke Thanks for the feedback.
I've updated the copyright and version info. I'm working on the feature UUID for the ATN serialization.
As for testing, I did include a test for the new command. Are you looking for something additional? (Such as testing ATN serialization/deserialization for the new command?)
Guys, I'm not able to add new features at this time...just a heads up. Especially something that would monkey with the state machines. sorry!
Maybe somebody else can maintain the repository? As I understand @ericvergnaud has privileges that allow PR merging. Otherwise, ANTLR development is stagnating a bit...
I don't have objections to such a feature, but tests that cover the new functionality are strictly required.
hi @KvanTTT well unfortunately a successful project is not about having somebody evaluate PRs for correctness and so on. ANTLR has been successful because I have been ruthless to my vision over the last 30+ years in the feature set and direction, so it's hard to just start letting people sticking features, such as this one. A project that works might stagnate as you say in terms of adding new features, but Microsoft Word was totally fine in 1995 and I'm not sure anything they added since has been useful to me. Sometimes products just work and sometimes they need new stuff. You'll notice that we have new targets, but I'm not gonna start adding new features simply because somebody thought they were cool.
Maybe it's time to start a fork and take over responsibility, if someone wants to do that. I fully understand Terence here, but on the other hand if a project doesn't evolve further, it's going to become obsolete over time. I'm pretty sure I wrote this before, but here we go: maybe it's time to hand ANTLR completely over to the community and let it develop to something which is no longer in your hands Ter? Or are there aspects we don't know, which would not allow that?
Forks are always welcome, for example Sam Harwell's. At this time I will maintain leadership even if it is going to asymptotically approach stagnation. People are always welcome to build a better tool, which is what I did because I didn't like yacc. There are lots of developers out there, not all of them make the right design and strategy decisions.
@parrt Is there a less intrusive way of incorporating this functionality that you would be willing to support?
From my perspective, the core change is allowing rewinding the lexer and rematching a token in a different mode. The goal being lexer lookahead in a way that is more flexible than predicates, and is not tied to a specific language runtime.
If not, I'll try to keep the branch up-to-date with master.
Thanks.
@dkleszyk Unfortunately I have to take into consideration all targets and any additional complexity and possible introduction of bugs etc. It is extremely hard to shepherd a project to success; one of the most difficult decisions a designer has to make is what to include and what to exclude. At this point I more or less excluding just about anything that touches the critical $ALL(*)$ algorithm. so in the end, I can't support this officially, certainly not just for one target.
Any way this can be reconsidered? Implementing lexers with ANTLR is way more difficult compared to JFlex.
