Is this project for ANTLR vs IDEA integration dead or alive

[gnom7]

Hey, I see in README it is recommended to use released 0.1 version:

Before 0.1, it was recommended to add this Git repo as a submodule of your own project, or to copy the source files directly.

It is now recommended to use the Maven dependency. The main breaking change is that the base package has been renamed from org.antlr.jetbrains.adaptor to org.antlr.intellij.adaptor

But since there is only one release, produced back in 2019 I'm wondering if it is still the recommended approach or should I rather include source in my own codebase? Project doesn't seem to be abandoned since I see some commits being added over time and 0.1 being released 5 years ago probably lags behind significantly by now.

Is there any resources/guides/documentation concerning ANTLR vs IDEA integration? I'd really prefer to use ANTLR but have hard time figuring out how to integrate it with IDEA/PSI w/o writing and supporting boilerplate code that could have been autogenerated otherwise (org.antlr.jetbrains.sample.psi package in jetbrains-plugin-sample). IDEA GrammarKit seems to generate lots of similar stuff automatically from grammar, but I'd much prefer to stick to ANTLR unless I'm really forced into GrammarKit due to lack of integration support. Is this project maintained? Are there any plans to improve IDEA integration support? How should I get the latest version of this project?

[bjansen]

Hi,

Using the Maven dependency is still the recommended approach. Nothing has been released in the last 5 years because nothing really changed for a long time. There's a few commits that are worth releasing though. I tend to put releases on hold because there's still a couple things that could be added to the next version, then I forget about them because I'm working on other things and I end up in this kind of situation 😅. I'm now trying to change that and release stuff immediately even if there are other things to do (better now than never), so I guess it's a good time to release a new version of this adaptor.

If you're familiar with ANTLR or if you're working on a language that has an existing ANTLR grammar, then this adaptor is the way to go. We don't generate code (yet), I guess it would be a bit overkill to generate one class per grammar rule (I don't remember if that's what GrammarKit is doing). Existing plugins like intellij-plugin-v4 and pebble-intellij use generic PSI classes and sometimes subclasses when needed. This allows for a pretty good IDEA integration in my experience.

[gnom7]

I wonder whether it makes sense to hookup into existing IDEA GrammarKit plugin to provide seamless ANTLR integration given GrammarKit is openly developed and used in JetBrains reference docs and tutorials. Providing ANTLR as and alternative to other 3rd party parser generators (JFlex) there probably would have made a lot of sense and better exposed ANTLR to audiences. Does it make sense? Are there any roadblocks (like I could imagine such integration might require ANTLR grammar extension to support similar PSI-related features GrammarKit grammar files support)? Why different approach have been chosen?

[bjansen]

JFlex only generates lexers, GrammarKit generates parsers that delegate the lexing part to JFlex, and ANTLR generates both a lexer and a parser. Moreover, ANTLR and GrammarKit each have their own grammar syntax and paradigm. ANTLR is a general purpose parser generator that supports multiple target languages, and GrammarKit is specifically tailored to generate PSI parsers for the IntelliJ platform.

[gnom7]

ok, so in such approach it would not be possible to make any use of existing GrammarKit infra/code? makes sense

thank you and looking forward for a new release of the adaptor :)

[gnom7]

btw do you have or plan some documentation on somewhat non-obvious parts of implementation like lexer state support etc? I'm poking with some toy plugin and getting 'Discontinuous sequence of tokens is generated by lexer', I suspect it is because I skip WS symbols in my lexer, am I right? Can I find about this in some documentation/tutorial? I don't mean to turn GitHub issues into an Q&A, sorry

[bjansen]

No, unfortunately there's no real documentation explaining what needs to be done to build a plugin based on ANTLR grammars. There's a plugin sample but it's outdated and the readme lacks info, ideally it should explain why the code does what it does.

You're right, whitespace is significant in PSI parsers, see https://plugins.jetbrains.com/docs/intellij/implementing-parser-and-psi.html#whitespace-and-comments

I don't mean to turn GitHub issues into an Q&A, sorry

Well Discussions is not enabled on this repo, so it's all right :)

[gnom7]

@bjansen hello again :) To address issue caused by skipped WS (-> skip) I initially tried to handle them explicitly at the lexer and the parser, which is cumbersome, but it worked. Then I checked plugin sample you referenced and redirected WS to HIDDEN channel, this resulted in all parse tree nodes being direct children of the root file node (flat, invalid tree), any clues of what I might be missing given such a symptom?

EDIT: alright, it looks like I have had to override getWhitespaceTokens in ParserDefinition and then handle new PsiWhiteSpace type everywhere in the code

[gnom7]

Is it possible to access labels of rule alternatives while using adaptor?

PSIElementTypeFactory.getRuleIElementTypes seems to handle rules themselves w/o any additional categorization of alternatives by labels etc.

Is there a recommended workaround? Should I refactor grammar to make alternatives a separate rules?

[mmeyer724]

Hello! It looks like the latest build published on maven is still v0.1 from five years ago. Any updates on pushing a new version? cc @bjansen

[bjansen]

Unfortunately I'm not authorized to publish Maven artifacts under the org.antlr namespace. Maybe @parrt can help?

[parrt]

I can't even remember which systems we need to push to or anything like that. Sorry. If somebody can tell me exactly what to do I can do it.

[bjansen]

Same as Antlr4, use mvn deploy (or the Gradle equivalent in this case) to upload release artifacts to Sonatype OSS, then go to their Nexus tool to "promote" the artifacts which will then be sent to Central.

In our case, I'd like to set up a GitHub Actions workflow to build and publish, but this means we would have to create GitHub secrets for the GPG keyring and its passphrase, not sure if this is acceptable for you (see https://central.sonatype.org/publish/publish-gradle/#credentials).

For the Nexus part (https://central.sonatype.org/publish/release/), either you or someone from the ANTLR team do it every time we release a new version, or you could add me to the list of "publisher users" in Sonatype's Central Portal I think (see https://central.sonatype.org/register/namespace/).

[parrt]

Trying now...looks like a gradle project though...

[parrt]

Ok, gradle can publish to mvn. won't build/compile. trying to update to later version of intellij lib. oh i need newer java now. fun.

[parrt]

Against all the odds, I think I figured it out:

beast:master:~/antlr/code/antlr4-intellij-adaptor $ ./gradlew publish

BUILD SUCCESSFUL in 13s
12 actionable tasks: 5 executed, 7 up-to-date

Hmm... It did not bump to 0.2, instead of just updated the snapshot. Is that OK?

https://oss.sonatype.org/service/local/repositories/snapshots/content/org/antlr/antlr4-intellij-adaptor/0.2-SNAPSHOT/antlr4-intellij-adaptor-0.2-20241109.185933-3.module

[parrt]

@bjansen I updated build to use nexus tokens and later java etc...

[bjansen]

Thanks, but would it be possible to add these tokens as GitHub secrets in order to release and publish automatically via tags and GitHub Actions?

[parrt]

Sure, though I've never done that. Can you make it easy for me and tell me exactly what to do?

[bjansen]

I added an Actions workflow to publish artifacts on OSS RH on each GitHub release.

It needs 4 secrets that can be configured like explained in Creating secrets for a repository:

• SONATYPE_USERNAME contains your OSS RH token
• SONATYPE_PASSWORD contains the password associated to this token
• SIGNING_KEY is the private GPG key used to sign artifacts, which you can export using this command:

gpg --export-secret-keys YOUR_KEY_ID_HERE | base64 > private.key

• SIGNING_PASSWORD contains the password for this private key

Then we can create a release, which will trigger the build, which should send artifacts to https://s01.oss.sonatype.org/. The last step is to "close" then "release" the repository like explained here (same as when you release antlr4).

[parrt]

OK, sorry about this but I spent a bunch of time trying to figure out what all that stuff meant. I think I went down the path of trying to create an OSS RH token at sonatype at one point and thought it was the most ridiculous process I've ever seen. I have a password at sonatype for publishing but I don't think that's what you need. I also don't know what the signing key is. Is that my private key for random stuff I do with gpg and maven or is that something specific to the OSS. Anyway, there's nothing worse than dealing with build systems and I'm just now working on this project anymore. sorry!

[bjansen]

I have a password at sonatype for publishing but I don't think that's what you need.

It is what I was talking about. The same password that you would use to sign in can also be used to automate publication.

Is that my private key for random stuff I do with gpg and maven

Yes, that's the one.

[parrt]

Hi Bastien, I just made you an administrator of that repository so I think you now have permissions to add whatever secrets we need. Hmm...actually, do you have permissions to the maven sonatype stuff? We can add you there as well so then you have complete control. Terence

On Tue, Dec 3, 2024 at 12:10 AM Bastien Jansen @.***> wrote:

I have a password at sonatype for publishing but I don't think that's what you need.

It is what I was talking about. The same password that you would use to sign in can also be used to automate publication.

Is that my private key for random stuff I do with gpg and maven

Yes, that's the one.

— Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4-intellij-adaptor/issues/47#issuecomment-2513816974, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABLUWLO34Y3SMMPU2A7OCD2DVRPBAVCNFSM6AAAAABIWA32ROVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKMJTHAYTMOJXGQ . You are receiving this because you were mentioned.Message ID: @.***>

-- Dictation in use. Please excuse homophones, malapropisms, and nonsense.

[bjansen]

Thanks. I have access to sonatype, but for my own projects at the moment. My username there is also bjansen. Maybe you can ask them to add me to the list of antlr contributors.

[parrt]

True to form, I literally can't understand anything dealing with their amazing software. https://help.sonatype.com/en/user-management.html looks nothing like the repository website I can log into. I see nothing that lets me log into sonatype. I just wasted half an hour and made no forward progress. I can't even find an email address to yell to. :(

[bjansen]

I figured out how to sign artifacts correctly, but now Sonatype is rejecting my attempts at pushing these artifacts in the org.antlr namespace:

Failed to publish publication 'maven' to repository 'maven' Could not PUT 'https://oss.sonatype.org/service/local/staging/deploy/maven2/org/antlr/antlr4-intellij-adaptor/debug/antlr4-intellij-adaptor-debug.jar'. Received status code 403 from server: Forbidden

I suppose ANTLR is still using the "legacy OSSRH", in which case permissions are managed by sending an email, according to https://central.sonatype.org/register/legacy/#add-or-remove-permissions-to-your-project

You will need to contact Central Support with the details of the request (i.e. adding or removing users), the groupId of your project and the username(s) in question.

In our case the groupId is org.antlr, the username is bjansen. You'll have to send the message from an email address that Sonatype knows, so I guess it's the one you used when you registered your account.

[mmeyer724]

Hello! In the meantime while this is being worked out, is there a way to directly use the latest version (or rather commit) of the adaptor in my plugin? i.e. pull directly from GitHub instead of from Maven Central

Apologies if this is a noob question, I'm just getting back in to Java development so I'm quite behind on how this all works now.

Best, Michael

[ris58h]

@mmeyer724 you can build the project locally and install it in your local Maven repository. Or install it in your company's artifact repository (like Nexus or Artifactory) if you use any. Another alternative is to use JitPack.

[bjansen]

@parrt do you know if someone from the ANTLR team could help us fix the release pipeline so we can publish to Central using Actions?

[parrt]

I’m really out of date on how to build or do anything with these tools. Is there a sequence of instructions I can execute that’s easy?

[ris58h]

I've created an issue for releasing 0.2 version in Maven Central BTW.