Redesign of the Node.js API Docs

[ovflowd]

Hey, y'all 👋 with the redesign of the Node.js Website done. We're ready to move our efforts into revamping the design of the Node.js API docs and its build process.

We understand this is code owned by Node.js Core, so we're (@nodejs/web-infra) opening an issue here.

Overview

Revamp of Tooling

• We intend to modernize the current tooling of the Node.js API docs (https://github.com/nodejs/node/tree/main/tools/doc), migrating from the current manual code into something more modern:
  • We intend to transform (in-memory, during build-time, no source change) the raw Markdown into MDX
  • Replace the YAML metadata (in-memory, during build-time, no source change) to references of MDX Metadata
    • This is similar to what we did with an MVP on the nodejs/nodejs.dev repository as seen here and here
  • The MDX gets converted into JSX, allowing us to use React Components on the codebase, such as on the Node.js Website
  • Finally, we use ReactDOM (Server) to compile the JSX into plain HTML (initial rendering), which is then embedded in the HTML templates, making each page an HTML file. (Source MD file -> Target HTML file as it has been done so far)
    • The file names will be the same
    • The JavaScript client-side code is then built for that specific page, tree-shaken, minimized, and stored on a target .js file
      • These JS files have a static filename but are linked within the HTML with a unique query param and hash to signal to the Browser that these are different versions.
    • All these output HTML and JS files are stored in the already defined out folder (as described on the make-doc Makefile step of the Node.js build process)
  • This revamp of the tooling allows more modern frameworks and libraries to be adopted.

Tooling Dependencies

• We aim to use preact as a JSX library (+ DOM rendering) since it's pretty much React but lightweight, it has pretty much 1:1 to React's API, and since the API docs are extremely simple statically generated page, we don't need anything fancier
• We aim to use Vite for the build process, transpiling/compilation and output to HTML files, and for the availability of HMR/Dev Server. This will be an experiment.
• We aim to use MDX to parse and transform the markdown into MDX/JSX.
• All other old dependencies would be removed/unnecessary. Some other devDependencies, like semver, might be needed.

Revamp of Design

After the revamp of tooling is done, we have a 1:1 feature to the old generation of API docs, and we can generate the same API docs with the same styles, same layout, and same components (the HTML snippets transformed into React Components), we can proceed with the revamp of styles.

This means we would adopt these designs based on the Node.js Website redesign into the API Docs. These designs may still change and are pending @nodejs/tsc approval)

This part is blocked by the monorepo transformation of the Node.js Website repository, which would allow all the existing components to be bundled into a UI components package.

We aim to use Shiki as we use on the Node.js Website for the CodeBoxes and Tailwind and Radix UI for Component tooling and a11y as we've done on the Node.js Website.

This would fundamentally course-correct the current design of the API Docs. Note that we aim to add a Search box (as we've done on the Node.js Website) and restructure the API Docs as described on this initiative of the Next-10 initiative.

It is part of the redesign of the Node.js API Docs:

• Applying the new styles, components, and UI as envisioned on Figma
• Better Navigation as envisioned on Figmas, which is doable by the newly revamped tooling (see the nodejs/nodejs.dev tooling attached above as a reference)
• Search input / Search boxes for the content of the API docs
• Better Table of Contents and overall reading based on the Figma designs

What is this issue about?

Keeping track of the efforts, communication, feedback, and progress of the revamp of the tooling and redesign of the API Docs is a sort of "Epic" for the whole initiative.

What's next?

Gathering initial feedback, consensus-seeking with Core Collaborators, and starting to delegate work on both fronts. Any support is welcome.

F.A.Q.

Are we going to translate the API docs?

No. It's an impossible and unmaintainable task. There is too much risk and work, and API docs get outdated fast. There is no merit in translating the API docs.

Can I help with the work on the API docs tooling?

Yes and no. We're more than open for feedback, ideas and code review, but the big chunk of work will be done by the Web Infra team which has a good knowledge of the tooling changes needed to be done and a good envisioning of what should be done.

Can I help with the redesign?

Yes! Implementation of components, designs, etc, is more than welcome! The actual components will reside on this repository and the Node.js Website. The API docs will only import the components/use them as the source will be on the Node.js Website repository.

Is there any timeline?

Not yet. We want to reach a consensus first and have a good understanding that the overall community is happy with these proposed changes + the proposed dependencies we want to use, such as MDX.

With MDX are the source files changing?

No. The transform into MDX will happen only in memory during the build time as a build-step. So that we can transform the non-conforming YAML metadata into something React can use. The source files will be untouched.

What are the problems with the current tooling?

• The current tooling does not support React.
• It has numerous limitations that hinder the implementation of desired features in the website redesign.
• The tooling is considered messy to update and not friendly for newcomers.
• It makes redesigning the API docs unintuitive and almost impossible without encountering significant blockers.

[aduh95]

Could you add an entry in the FAQ explaining why this is necessary / what the current format is blocking please?

[ovflowd]

what the current format is blocking please?

What current format? This issue doesn't aim changing the format of the Markdown files

explaining why this is necessary

Do you mean why a redesign is necessary?... Isn't it obvious? (I can write it down of course, but just wondering what you're expecting me to write xD)

[aduh95]

We have a current toolchain that takes our Markdown(+YAML) files and output HTML, this issue is suggesting introducing a different toolchain that would basically do the same. Maybe a better way of phrase "why is that necessary?" would be "why not redesign the docs using the current toolchain?".

[ovflowd]

We have a current toolchain that takes our Markdown(+YAML) files and output HTML, this issue is suggesting introducing a different toolchain that would basically do the same. Maybe a better way of phrase "why is that necessary?" would be "why not redesign the docs using the current toolchain?".

We cannot use the Website redesign with the current tooling. The stacks are entirely incompatible. For starters, the current tooling doesn't support React. The initiative here is to modernize the tooling (pretty much from scratch, based on our successful experiment on nodejs.dev) so we can use React.

The current tooling has numerous limitations that would make implementing all the things we want with the redesign impossible.

For the average Node.js core collaborator, the new tooling wouldn't require an extra lift to contribute to API docs or their templates. Styles would still be pure CSS (with the addition of Tailwind), and the website team would maintain the UI components on the Website repo.

Layouts/Templates would be in JSX, while the base template is still in HTML. The revised tooling would be extra documented and use standard software design practices. Unfortunately, the current tooling is a bit messy to update and not friendly for the average newcomer.

We would also be able to reuse a lot of the work we did on the Node.js website, for example, with a Rehype Plugin for Shiki (the Syntax Highlighter) that would allow us to do the style of code blocks done on the API Docs, effortlessly.

The base of the logic (use this just as a reference, as a lot would be updated and removed) would be https://github.com/nodejs/nodejs.dev/blob/main/util-node/apiDocsTransformUtils.js, which is cleaner than our current tooling.

In other words, we want to modernize the tooling and add modern technologies that would allow us to share similar stacks with the Node.js Website and reuse the work we've done there. Unfortunately, redesigning the API docs with the current stack is unintuitive and almost impossible without significant blockers.

[joyeecheung]

Can we move the external dependencies and tools to a different repo, and only keep the markdown documents in this repo? I imagine the new dependencies and build process could make the already flaky CI even more flaky and complicate releases and backports further if they end up in this repo (my brain already starts to hurt when thinking about backporting the tool changes to v18 and then make the addon docs build on SmartOS).

[ovflowd]

Can we move the external dependencies and tools to a different repo, and only keep the markdown documents in this repo? I imagine the new dependencies and build process could make the already flaky CI even more flaky and complicate releases and backports further if they end up in this repo (my brain already starts to hurt when thinking about backporting the tool changes to v18 and then make the addon docs build on SmartOS).

That is definitely doable. Didn't add to the initial proposal as Im not sure how the feeling/consensus about this one would be.

But I definitely would say a +1 to have the tooling somewhere else. The only issue is, generating the docs is part of Node.js build process (Makefile) so Im not sure how we should handle that transition? Would node core need to depend on another node repository and add it as part of the build process? That is currently out of my expertise but I can definitely 👀👀 into it!

[AugustinMauroy]

+1 for proposal if we do something like node-core-utils but on nodejs.org it's should work ?

[bmuenzenmeyer]

That is currently out of my expertise but I can definitely 👀👀 into it!

this would be possible either with direct github install instructions or npm publication - both pretty strightforward

[joyeecheung]

But I definitely would say a +1 to have the tooling somewhere else. The only issue is, generating the docs is part of Node.js build process (Makefile) so Im not sure how we should handle that transition? Would node core need to depend on another node repository and add it as part of the build process? That is currently out of my expertise but I can definitely 👀👀 into it!

We can still keep a basic renderer here just to make sure that the docs are parsable, and also we still need to generate addon tests from addons.md.

[mcollina]

@ovflowd I think docs must "render" in the browser without JS enabled. Ideally no per-page JS. (Additional features should use JS, such as search).

[ovflowd]

@ovflowd I think docs must "render" in the browser without JS enabled. Ideally no per-page JS. (Additional features should use JS, such as search).

Definitely agreed here, IMO output of both should be a 1:1 match.

[joyeecheung]

Not sure if this is the right place to mention it, but during the summit I think several people mentioned in different sessions about our docs being too intimidating to beginners - I think by that they mean, API docs are like a dictionary, and you don't learn a language by reading a dictionary. We should have more tutorials in the docs before branching into the API dictionary. Which may affect the redesign because the page layout and information architecture etc. need to consider this.

[ovflowd]

Not sure if this is the right place to mention it, but during the summit I think several people mentioned in different sessions about our docs being too intimidating to beginners - I think by that they mean, API docs are like a dictionary, and you don't learn a language by reading a dictionary. We should have more tutorials in the docs before branching into the API dictionary. Which may affect the redesign because the page layout and information architecture etc. need to consider this.

I believe that's why we're cultivating an improved Learn section on nodejs.org :) I believe the API docs should be more an API reference, but agree they could be rewritten, improved, and have more examples... But that's not the scope of this issue.

[richardlau]

https://documentation.divio.com/ is an interesting POV on documentation -- our API docs fall into "reference". The main point being that there are different types of documentation and documentation is clearer if those are not mixed together.

[legendecas]

(Removing label https://github.com/nodejs/node/labels/node-api since this is not specific to the native c api)

[ovflowd]

documentation.divio.com is an interesting POV on documentation -- our API docs fall into "reference". The main point being that there are different types of documentation and documentation is clearer if those are not mixed together.

Right, I believe my goal here is just a redesign. I do agree that content-wise our API docs should fall into "Reference" documentation; And we should write Learn material (on the Node.js Website) with actual learning materials for our API Documentation.

This is possibly something we want to invest time into.

For example @JakobJingleheimer was working on learning material for Node.js Loaders :)

[JakobJingleheimer]

For example @JakobJingleheimer was working on learning material for Node.js Loaders :)

Indeed 🙂 the first draft is nearly ready (have to first write a package to facilitate an important use-case needed in thr article, which is nearly ready).

[ovflowd]

Hey, @nodejs/tsc I'd like to pin this issue, due to its high-visibility and impact on the project.

Throughout the years, one of the most common "feedback" factors we've received on social media have been the current state (design) of the API docs; I believe as we're ramping up to work on this enormous piece of work, I'd make sense to highlight the visibility of this issue:

• Better reach to our outer and inner community
• Gather feedback fast and make announcements easy for people to follow

[benjamingr]

Hey, @nodejs/tsc I'd like to pin this issue, due to its high-visibility and impact on the project.

Pins are typically used in order to catch cases we have breakages/bugs in order to prevent users from opening duplicate issues. I'm not sure what pinning would accomplish here?

[ovflowd]

We have an update: We finally started the work on the new API Doc Tooling, and it's getting nicely done.

The initial PR (https://github.com/nodejs/api-docs-tooling/pull/33) allows us already to have a fully working parser for the API Docs and will enable us to start the following work to add the HTML and JSON generators.

[ovflowd]

We've landed the core of the new API doc tooling and are proceeding with the generators that will allow us to achieve what we need.

[ovflowd]

We've landed the CLI pieces of the new API doc tooling. We're moving forward with implementing the generators that allow us to generate the JSON, HTML and whatnot outputs of the API docs.

You can try the CLI already by doing:

npm i -g https://github.com/nodejs/api-docs-tooling/tarball/main

Then clone nodejs/node, (for example gh repo clone nodejs/node);

And then run the api-docs-tooling CLI, for example:

api-docs-tooling -i /path/to/node/node/doc/api/http.md -t json-simple -o .

Note that this is all alpha; there will be many changes yet! Also check api-docs-tooling -h for help :)

[ovflowd]

As https://github.com/nodejs/api-docs-tooling/pull/55 is getting closer to land and @flakey5 is going to work on the JSON generator; I believe we should soon start seeing how we'd replace the current code within tool/doc and other places to swap out the existing API doc generator towards the new one.

[ovflowd]

FYI the legacy HTML generator for the new API doc tooling is ready and feature complete https://github.com/nodejs/api-docs-tooling/pull/55