Automatic TypeScript definition generation from JSG RTTI

[mrbbot]

(re-do of https://github.com/cloudflare/workerd/pull/110, let's try this again 🙂 )

Hey! 👋 This PR adds support for generating TypeScript types from JSG RTTI, replacing the internal autodecl script. This forms the basis for the next workers-types version. These scripts are located in this repository rather than workers-types as they depend on Bazel outputs, and we'd like to be able to share the Bazel cache in CI. Going forward, DevProd should probably be the CODEOWNER for everything in the types directory.

To generate types, run:

$ bazel build //types:types

A gist containing the generated types can be found here. This also includes a copy of the types before TypeScript transformers are applied.

(fyi, about half of the additions are from pnpm-lock.yaml)

Implementation Notes and Questions

1. Added a jsg::fullyQualifiedTypeName() method. This behaves like jsg::typeName(), but includes namespaces and template arguments in the returned name. Namespaces are required to differentiate nested types with the same name (e.g. DurableObjectStorageOperations::GetOptions, KvNamespace::GetOptions and R2Bucket::GetOptions). Template arguments are required to differentiate jsg::IteratorBase... types generated by JSG_ITERATOR.
2. In workerd::jsg::rtti::Builder, replaced the symbol key from jsg::typeName() to jsg::fullyQualifiedTypeName() for the reasons above. This also affects which values should be passed as parameters to workerd::jsg::rtti::Builder::structure().
3. Changed the rtti.capnp schema. Generally, I tried to evolve the schema in a backwards-compatible way, but it would be cleaner if we're still able to make breaking changes. /cc @mikea
   1. Added fullyQualifiedName fields to Structure and StructureType schemas for the reasons above. I was hesitant to make name fully-qualified since I wasn't sure which other code depended on RTTI.
   2. Added support for jsg::LenientOptional to RTTI. When generating TypeScript definitions, we also need to be able to distinguish between optionals expecting null (kj::Maybe) and others expecting undefined. A new name field has been added to the MaybeType schema, similar to NumberType and StringType.
   3. Similarly, we need to be able to distinguish between kj::Array, kj::ArrayPtr and jsg::Sequence, so a new name field has been added to the ArrayType schema. It may be better to make this and the previous maybe field enums instead?
   4. Added a name field in a group with nested members. This ensures members coming from JSG_NESTED_TYPE_NAMED macros have the correct names.
   5. Added iterator and asyncIterator fields to the Structure schema. Whilst there are already iterable and asyncIterable boolean fields, we need to know the full method types (especially the returned (Async)Iterator type) for [Symbol.iterator]/[Symbol.asyncIterator].
4. Added a new api-encoder.c++ entrypoint that spits out RTTI to a file. All TypeScript generation scripts are written in TypeScript, so we can use the official TypeScript compiler API for creating/processing/printing AST nodes.
5. Setup Bazel to build/run JavaScript/TypeScript programs. I used https://github.com/aspect-build/rules_js instead of https://github.com/bazelbuild/rules_nodejs as this seems to be the actively maintained version. Note aspect-build/rules_js requires us to use pnpm. We should be able to use this setup for packaging/publishing workerd npm packages too. /cc @penalosa

TODOs

(to follow in later PRs)

1. Overrides: whilst correct TypeScript types are being generated without them (an improvement on autodecl 😅), they're not as accurate as they could be.
2. Parameter names: currently parameters are named param0, param1, ... . Ideally, we'd use the actual C++ parameter names here, so we need some way of getting these into the RTTI.
3. Compatibility dates: we currently generate a single set of types with all non-experimental compatibility flags enabled. We'd like types to depend on which flags users have enabled.
4. CI: we'd like to build/publish types automatically on PRs/releases.
5. Multiple files: we currently generate a large single file containing all types. We'd like to split this up into multiple purpose-related files, hence this implementation splits structures into named groups.

[jasnell]

@mrbbot ... ok, so the reason we had to revert this was twofold:

1. The bazel changes here also require bazel changes for the internal platform, and
2. The code changes here cause compile failures on the internal platform.

@mikea also needs to review the code changes here before it can land. The changes here LGTM but Mike is the domain expert so while I might sign off it should wait to land until both of the following happen:

1. Mike has a chance to review this and sign off, and
2. There is a corresponding internal PR that updates the internal platform and that has a successful CI run.

[penalosa]

This may have been fixed in newer commits, but slight bug in the version you posted; everything URL related is prefixed with URL. So, for instance, the URL type is urlURL.

[mrbbot]

everything URL related is prefixed with URL. So, for instance, the URL type is urlURL.

@penalosa we'll fix this when we add overrides that can rename types. 👍 The issue is that the spec-compliant URL implementation is under an additional url:: namespace so as to not conflict with the original. We should only be including one of these sets of definitions, so renaming the url:: types back to their spec-names shouldn't be an issue.

[JacobMGEvans]

Exciting.

[jasnell]

It appears that all of @mikea's feedback has been addressed. I'll clear their "change requested" now.

[jasnell]

@mrbbot this should be ready to land. Once landed, remember to rebase the matching internal PR on workerd main and let the CI run again there to confirm it's ready to land.