EFI: Publish and maintain types

[sheplu]

Motivation

Types are widely used in the ecosystem, and requested as shown in the Next-10 survey last year. By providing them directly (or helping the community to provide them) would be a good help for the community. But this can be time consuming and not be perfect with versionning (based on typescript)

Expectation

Evaluate if we want to provide more type documentation using TS or JSDoc (or other solution)

Implementation

Status

Part: Technical

Draft

Types of the Express ecosystem are “at best” available and working. But they can be greatly improved. By automatically building and publishing types when we release a new version it would greatly help users. The community at scale rely more and more on typing (or the equivalent using JSDoc) and we should work to publish them automatically. Add types inside package Publish dedicated package

[UlisesGascon]

As far I know currently many people use https://www.npmjs.com/package/@types/express. Do we plan to contribute there or just integrate this into our code base?

IMO, anything that we can do to help our community to use Express is a good move, but I don't see a big value as maintainers to rewrite to TS as we already have JSDocs in place (for us and for the community). Also I am not very use to work with TS, so maybe all the plan is smoother than I imagine.

[dougwilson]

Historically we have not provided types in the module packages bc 1) almost none of the TC nor the what we would call project captains were interested in writing TypeScript (I for one am not interested at all in writing or maintaining any Typescript l, as I am here for the Javascript and the typeless language) and the bigger point 2) how to actually deal with semver as Typescript still on the regular makes new releases that break existing types in ways they need to be updated, which then breaks them on old Typescript and users get fustrated and the project has to make a bunch of weird releases, which the @types packages have the issue since they are separately versioned.

[timodempwolf]

Is there any advantage of using TS instead of JSDocs? As far as I am aware tsserver works as well with JSDocs as it does with Typescript types, but I'm no expert on the subject.

[wesleytodd]

Do we plan to contribute there or just integrate this into our code base?

We are in a bit of a rock/hard place here. Because of their stance on semver I don't think we can really do either well. Since we are strongly committed to semver support, I almost think it is ideal that we take a hands off approach. That said, if we had folks who were really passionate about it and could make a clear case one way or another I would love it. Mainly because I agree that "anything that we can do to help our community to use Express is a good move".

how to actually deal with semver as Typescript still on the regular makes new releases that break existing types

Yep, this is fundamentally the problem. If we take ownership of them it become our problem. And while I agree we should do what we can to help folks using Express, I think a better way would be to JSDoc things and then let the folks on the TS/DT side generate their types from our JSDoc than to actually ship or build the types our self and deal with their bad decisions around semver.

[wesleytodd]

As I am thinking about this, maybe the way we could do this "best" is the following:

1. Add robust JSDoc type annotations to all our packages
2. Generate a .d.ts file and run a subset of tests against it (tests to specifically validate the types, not the behavior), but don't ship that for users
3. When those get updated, have some automation which pings someone from DT (need them to agree to that obviously) so they can update

[wesleytodd]

I was going through cleaning up other threads in here and found this one: https://github.com/expressjs/discussions/issues/7

[ljharb]

The major advantages I see of sticking to DT instead of inline types are:

1. no semver conflation - the JS package doesn't get bumps only for type changes and vice versa, and breaking changes in types don't have to be major bumps in the packge
2. the TS team helps manage TS compatibility - it's complex to define what TS versions are supported and support multiples, and the DT repo/TS team just handles this

[fdrobidoux]

2. the TS team helps manage TS compatibility - it's complex to define what TS versions are supported and support multiples, and the DT repo/TS team just handles this

This shouldn't be an issue for legacy software. Active development would use a local version of TS instead (e.g. bundled with VSCode) even with legacy express apps.

[wesleytodd]

@fdrobidoux I don't follow. Can you explain what you mean? The issue @ljharb is talking about is that if we publish types along with our packages (instead of relying on DT) we then have to deal with all the version compatibility issues. If we let them handle it, they deal with it. Are you trying to say something else?

[ljharb]

Some extra commentary (some of which already exists here) in #203.

[wesleytodd]

https://github.com/pillarjs/router/pull/100 https://github.com/pillarjs/router/pull/110 https://github.com/pillarjs/router/pull/107

I was cleaning up the router issues today and came across these. Do we think we need to decide on this before v5? I am still on the fence. I was going around asking folks if they wanted to work on landing things asap but I don't want to promise this unless we have a clear decision on doing this in repo or with DT.

[rafaell-lycan]

My 2 cents on this:

TypeScript brings better DX overall; JSDoc does not bring proper types IMO.

I'd rather have it on the same package if possible, like Fastify did, but devs already know that Express doesn't ship types anyway.

If the TC commits to keeping @types/express up-to-date on the next versions of Express with the community I don't see a problem.

[ljharb]

jsdoc checked by TS, with handwritten d.ts files, gives the same quasi-correctness that TS does.

[wesleytodd]

quasi-correctness

this is key.

JSDoc does not bring proper types

And neither does typescript if that is a thing you care about.

If the TC commits to keeping @types/express up-to-date on the next versions of Express with the community I don't see a problem.

We need a champion for this. If we have that I am no opposed at all to maintaining the types (either in package or in DT, depending on what the champion decides). The problem is we have never had anyone who signed up to do that. And to be clear, I don't mean signing up to re-write it in TS. I would block that or leave the project if there was consensus and I was the only objector.

[broofa]

[Edit: My apologies for the long comment. I recognize that I'm probably not moving the conversation forward all that much, and my conclusion is pretty much the same as that of @wesleytodd. I.e. That this needs a champion for it to be meaningful. This is just me processing my own thoughts on this issue, which I've dealt with numerous times in other projects.]

'Would love to see this made a priority but I fear it's a bit intractable. The problem is you can't really talk about supporting TS users without [eventually] accepting that the conversation has to be about porting a project's codebase to TypeScript.

Is that making a mountain out of a molehill? Yeah, somewhat. But this is a slippery slope that goes something like this...

Step 1: "La la la la. Not our problem!"

"Hey, we can just let "the community" solve this for themselves. That's what the Definitely Typed™ project is for!"

Nope, doesn't work. Oh, sure, @types/express will work for 90% of users. But there will always be small issues and inconsistencies that affect that long-tail 10%, especially if this express switches to a regular release cycle, which seems to be the plan. 'Problem is, the only place for those issues to get reported is here, in the express project. But good luck getting the @types/express ✌️ maintainers✌️ to step up and help out here. (Note: to the extent they do, we've moved on to Step 2 ...)

Step 2: "Okay, fine. It's our problem."

The type-related issues users file is a constant reminder that the buck actually does stop with express when it comes to TS types. So express is eventually going to have to accept responsibility for maintaining the DT types. But manually crafting types is a lot of work, as is meeting the DT project requirements for publishing new releases. Thus, this tends to be a short-lived solution that leads to a) tooling to automate creation of types, and b) bundling types with your project.

Farewell, Definitely Typed. We hardly knew ye.

Step 3: "Pretend we're a TS-project w/out actually being a TS project"

** sigh ** Fine, we'll bundle type declaration files with our project. And we'll somehow autogenerate them. But how?

Oh hey, look, JSDoc!

JSDoc sort of works. It's certainly tangibly better than steps 1 or 2 for a project like express (maybe?) But it just moves the manually-crafted nature of the types out of the *.d.ts files and into source comments. And JSdoc is it's own, different language that not all devs are familiar or comfortable with.

Ultimately, the problem with all three of these options is that type-related issues tend not to get discovered until after a release has been published. Fixing them requires more releases, leading to unnecessary code churn.

Step 4: "Actually become a TS-project"

... and so we arrive at the final form of the types problem.

Porting the project to TS kills two very big birds with one stone:

1. Solves the type declaration problem for real. Transpiling TS -> JS gives you type declarations for free. Problem. Solved.
2. TypeScript results in higher-quality code. Static type checking catches a plethora of errors that linting and unit tests simply don't cover. This is particularly important for an opensource project that takes as many drive-by contributions as express.

To my mind, debating the extent to which Definitely Typed should be involved or how/if JSDocs should be used is kind of postponing the inevitable. The only way of staving this off is if someone is willing devote significant time and effort to maintaining types by hand. Is someone here willing to be that person?

(Full disclosure: personally, I would view anyone volunteering for that with a bit of skepticism. This can't be one or two-off "sure, I'll do that" thing. They should be willing to be "on call" for type-related issues for the foreseeable future.)