Add projects to test typescript support with different compiler options

[JesusTheHun]

Following #4218, this PR adds three test projects that use ESM and CJS with npm and one that uses ESM with pnpm.

The projects are compiled with different typescript compiler options to ensure compatibility.

This PR focuses on testing that antlr4 can be imported in TS projects. This is why it does not include non-TS test projects.

[ericvergnaud]

Looks like the tests fail ? See for example https://github.com/antlr/antlr4/actions/runs/8723991967/job/23933679496?pr=4597

[JesusTheHun]

Looks like the tests fail ? See for example https://github.com/antlr/antlr4/actions/runs/8723991967/job/23933679496?pr=4597

I fail to run the tests locally using maven. I haven't touch to java in a while so I'm a bit rusty at identifying compilation chain issues 😅 In addition to the requested changes I've committed a fix, hopefully the one we need.

[JesusTheHun]

Fixes the failing tests, but not sure it satisfies the requirement to test against the built package (vs the source project)

I was hesitant to test against the build because that would require to compile every time you make a change, which is annoying during development. But I can make the change if you want. It is safer indeed.

EDIT : your build process bundle the javascript, but do not include the type definitions. So we would have to use npm pack, extract the archive, and use the directory at the package path. That being said, what is tested really is the declaration of types definition, which only targets the package.json really. Assuming the referenced files exist. Before my latest commits they did not, and the build failed. So I say the tests served their purpose 👍

[JesusTheHun]

@ericvergnaud just pinging in case you didn't see my edit above

[ericvergnaud]

@ericvergnaud just pinging in case you didn't see my edit above

Ah I see what you mean. But how can we be sure that your test samples are using the dist folder ?

[JesusTheHun]

Ah I see what you mean. But how can we be sure that your test samples are using the dist folder ?

The test uses the package.json. The package.json references the dist folder for the javascript, and the src for the type definitions.

EDIT : to clarify, npm pack is the command that create the archive that is published. It copies the package.json as-is, so our tests have the exact same conditions.

[JesusTheHun]

@ericvergnaud after migrating the project using this lib to pnpm, I noticed the types stopped to work altogether. After looking around I noticed that the imports inside the definition files did not include the file extension, which is required for modern node usage.

EDIT : for anyone following this topic, I've published a package that reflect the changes on this branch : https://www.npmjs.com/package/@cbjsdev/antlr4 . Install the latest version 4.13.2.

[ericvergnaud]

Hi, sorry for the delay. I suspect that the reason you need the .js extensions is because the tests are running against the source code rather than the built package. Node does not support TypeScript, and should be using the web packed module. I suggest you revert commit 40bb02cb53137ea6e4aee3a43488729d90d84f05, which makes this PR way too broad.

[JesusTheHun]

@ericvergnaud The need for the extension comes from the node16 TypeScript target. TypeScript uses the d.ts files, so it's unrelated to the packaged source files.

[ericvergnaud]

@JesusTheHun sorry but I don't follow:

• I've been using these .ts files "as is" for years now without any problem
• Typescript uses the .ts files, and doesn't need the .js extension
• node16 is unaware of these .ts files, and should be using the webpacked module

What am I missing ?

It seems the issue appears when introducing pnpm ? That's a separate topic altogether. Can we restrict this PR back to its primary intent ?

(FWIW we are preparing a release, I'd like to get some of this in)

[JesusTheHun]

@ericvergnaud When I mention node16 I'm talking about the TS compiler option "target". Not the node runtime version.

pnpm is the most strict package manager. It enforces standards and prevent bad usage instead of working around them. This may be the reason why only pnpm issues errors.

The intent of the PR is to make sure projects can import Antlr without issue, right ? If so, the commit is required, otherwise pnpm users won't be able to import Antlr.

Also, I'm on holidays without my machine. So I wont be able to revert the commit myself before August 5th.

That being said, I'm curious about the reason for rejecting the commit. Ok it makes a lot of file changes, but they are trivial changes, it's not like they each require a review.

[ericvergnaud]

The reason is that I'm in favor of small steps. We currently don't support pnpm and I don't need that additional responsibility (similarly we only support webpack). So let's keep this PR as small as possible and ensure it covers the currently supported environments:

• node >= 16
• typescript >= 4.8
• webpacked code from the dist folder

[JesusTheHun]

I understand the need for small steps.

pnpm falls into the support for node >= 16, since it's officially distributed by Node.js via corepack.

I understand the commit touches a lot of files but the changes are tiny and identical in each file. I don't see any drawback.

[JesusTheHun]

@ericvergnaud I'll be back in a few days, can you hold the release until then ? I assume you also want me to delete the test project for pnpm ?

[ericvergnaud]

@parrt not sure what is your ETA for the release ?

[ericvergnaud]

@ericvergnaud I'll be back in a few days, can you hold the release until then ? I assume you also want me to delete the test project for pnpm ?

Yes please rollback pnpm related changes

[parrt]

I’m gonna try for this weekend maybe tomorrow

Dictation in use. Please excuse homophones, malapropisms, and nonsense.

On Fri, Aug 2, 2024 at 12:03 AM Eric Vergnaud @.***> wrote:

@parrt https://github.com/parrt not sure what is your ETA for the release ?

— Reply to this email directly, view it on GitHub https://github.com/antlr/antlr4/pull/4597#issuecomment-2264707001, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABLUWIDCTL6R6UWDCNRVADZPMVLPAVCNFSM6AAAAABGLOZCY2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENRUG4YDOMBQGE . You are receiving this because you were mentioned.Message ID: @.***>

[ericvergnaud]

@JesusTheHun hi there, it's been a while... Any progress ?

[ericvergnaud]

@JesusTheHun Thanks, this is looking good. The TS tests fail though. Could you take a look ?

[JesusTheHun]

@JesusTheHun Thanks, this is looking good. The TS tests fail though. Could you take a look ?

This looks unrelated to this PR, as a previous action failed the same way. See https://github.com/antlr/antlr4/actions/runs/10863756846/job/30148277515

[ericvergnaud]

@JesusTheHun Can you rebase ?

[JesusTheHun]

@ericvergnaud all clear ☝️

[ericvergnaud]

@parrt blessed @JesusTheHun Many thanks for this !!!