MDX 3

[kachkaev]

MDX 2 was released on 1 Feb 2022: https://github.com/mdx-js/mdx/releases/tag/2.0.0 UPD: MDX 3 was released on 24 Oct 2023: https://github.com/mdx-js/mdx/releases/tag/3.0.0

The changes are summarised in the blog post: https://mdxjs.com/blog/v2/ UPD: https://mdxjs.com/migrating/v3/

Here is a big one for Prettier:

Originally, the format was very close to how markdown, and HTML in markdown, works. We found that the old behavior often yielded unexpected results. In version 2, we shift the format a little bit closer to how JS(X) works.

The example below is from the blog post. It is a valid MDX in v2, but Prettier 2.5.1 is unable to parse it:

Prettier 2.5.1 Playground link

--parser mdx

Input:

<div>*hi*?</div>

<div>
  # hi?
</div>

<main>
  <div>

    # hi?

  </div>
</main>

Output:

SyntaxError: Unexpected token (2:12)
  1 | <$><main>
> 2 |   <div></$>
    |            ^

Expected behavior:

No syntax error

---

Let’s track MDX ~2~ 3 support here 👀

[sosukesuzuki]

Remark and MDX ecosystems use Pure ECMAScript Modules Packages. We need to use that parser, and I think the our ECMAScript Modules migration(#10157) block this work.

[sosukesuzuki]

For now, As a first step, I will restart the migration to remark-parser v9 (#11938)

[wooorm]

As you’re already bundling prettier when releasing, could you bundle mdx-js/mdx / remark-parse and such as CJS?

[sosukesuzuki]

Do you mean bundling ESM Packages(mdx-js/mdx, remark-parser) into CJS(bundled prettier)? It may be possible, but it will no longer be possible to run the source code as-is in Node.js, as is current Prettier for development.

/cc @fisker

[wooorm]

but it will no longer be possible to run the source code as-is in Node.js

I’m suggesting something like an npm script that works like this:

npx esbuild @mdx-js/mdx --bundle --platform=node --outfile=vendor/mdx.js

That could be run once in a while. Like, every month? It doesn’t have to affect running Prettier in development?

[sosukesuzuki]

Thank you! I think we can use the way you suggested. It may solve the problem that we cannot use other Pure ESM packages I'll try it.

[dburrows]

Just to add, when the parsing does work, the inline markdown within components in MDX2 is mangled - see the Playground example here

Input

<div>
  ## foo
  
  a paragraph
  
  another paragraph
</div>

Output

<div>## foo a paragraph another paragraph</div>

Expected output

<div>
  ## foo

  a paragraph
  
  another paragraph
</div>

Not sure whether the parser update will fix this.

[sebastienbarre]

Thank you people for working on this. If this helps, here is another problem I'm experiencing in a MDX file which I was just told is related to this very issue:

MDX also supports JavaScript expressions inside curly braces, i.e.: {Math.PI * 2}

Prettier automatically changes it to:

MDX also supports JavaScript expressions inside curly braces, i.e.: {Math.PI \* 2}

Which breaks MDX parsers -- this is no longer a valid Javascript/JSX expression and it won't be executed.

Using Prettier's recommended <!-- prettier-ignore --> won't help because it breaks the MDX parser as well (because of the < used for React/JSX components), and using MDX's recommend {/* this is a comment */} doesn't work because it is not recognized by Prettier since the file is treated as Markdown (as it should), not JSX.

Thank you.

[sosukesuzuki]

we unpin this issue temporary because we should pin #12931

[JounQin]

I'd like to work on this, maybe this weekend.

For my experience to add support for mdx@v2 at eslint-mdx (see https://github.com/mdx-js/eslint-mdx/pull/284), it should not be quite tough.

Or maybe I should wait for #13201

cc @fisker

[fisker]

That's great! The difficult one should be upgrade remark.

Or maybe I should wait for https://github.com/prettier/prettier/pull/13201

That's for prettier plugin, I don't think you need feature from that. But we can merge if you want.

[JounQin]

The difficult one should be upgrade remark.

@fisker Why do you think so?

[fisker]

There are already at least 3 people (including me) tried. But maybe we just don't have enough experience. Looking forward to see you start work on it.

[fisker]

FYI: There is still an open one https://github.com/prettier/prettier/pull/11867

[JounQin]

Personally, I'd like to use something like https://github.com/un-ts/synckit to support ESM packages like I did at https://github.com/mdx-js/eslint-mdx/pull/284, instead of hacking bundling which could be very verbose like https://github.com/prettier/prettier/pull/11867/files#diff-1e058ca1442e46581b13571fb8d261f9e1f5657e26c96634d4c1072f0f0347f1R60-R76.

Is that acceptable?

---

It seems we've migrated to ESM, so synckit or hacking is not needed here.

[JounQin]

https://github.com/prettier/prettier/blob/main/scripts/vendors/README.md

It is still interesting, @fisker do we still must bundle ESM packages?

[fisker]

No, checkout the next branch. https://github.com/prettier/prettier/pull/9583

[fisker]

@JounQin Progress?

[JounQin]

@JounQin Progress?

Ah, I'm a bit busy these days, life and business at the same time. Hope I could find more time on prettier next week.

[kachkaev]

I created a discussion in the MDX community asking for some help: https://github.com/orgs/mdx-js/discussions/2127. Shame that I don’t have spare capacity to work on this, but I’d love to see MDX 2 in Prettier v3 🤞

[JounQin]

@fisker

I'm just trying to start the work at https://github.com/JounQin/prettier/commit/1554e63b1f6d56b63714d2860e779546e8a02a34, but I cannot even make the test work.

yarn test

Usage Error: Couldn't find the node_modules state file - running an install might help (findPackageLocation)

$ yarn run [--inspect] [--inspect-brk] [-T,--top-level] [-B,--binaries-only] <scriptName> ...

No idea what is happening.

[fisker]

Do you have nodeLinker configed somewhere?

[JounQin]

Do you have nodeLinker configed somewhere?

No. And the project's .yarnrc.yml is

defaultSemverRangePrefix: ""

enableGlobalCache: true

nodeLinker: node-modules

yarnPath: .yarn/releases/yarn-3.2.3.cjs

[fisker]

I have no idea. Have you try reinstall deps?

[JounQin]

I have no idea. Have you try reinstall deps?

OK, it seems installation is throwing:

➤ YN0000: ┌ Resolution step
➤ YN0002: │ prettier@workspace:. doesn't provide @typescript-eslint/parser (p4df2b), requested by @typescript-eslint/eslint-plugin
➤ YN0002: │ prettier@workspace:. doesn't provide rollup (p291e4), requested by rollup-plugin-license
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 1m 7s
➤ YN0000: ┌ Fetch step
➤ YN0013: │ parse-srcset@https://github.com/ikatyang/parse-srcset.git#commit=54eb9c1cb21db5c62b4d0e275d7249
➤ YN0013: │ parse-srcset@https://github.com/ikatyang/parse-srcset.git#commit=54eb9c1cb21db5c62b4d0e275d7249
➤ YN0018: │ parse-srcset@https://github.com/ikatyang/parse-srcset.git#commit=54eb9c1cb21db5c62b4d0e275d7249516df6f0ee: The remote archive doesn't match the expected checksum
➤ YN0000: └ Completed in 5s 364ms
➤ YN0000: Failed with errors in 1m 13s

I found https://github.com/yarnpkg/berry/issues/1142#issuecomment-607272809, it's working now.

[remcohaszing]

Ah, I'm a bit busy these days, life and business at the same time. Hope I could find more time on prettier next week.

I would love to see this happen too! Let me know if you need any help.

[remcohaszing]

Because @JounQin is busy, I’m now working on this. Instead of using customized logic to parse the MDX specific syntax, I’ll use remark-mdx.

[remcohaszing]

Small update: In order to fix MDX 2 support, I first need to update unified and remark. I discovered a couple of corner cases where markdown formatting will change a little, but in more correct ways. I’m currently in the process of fixing some new issues that arose that are definitely wrong.

[remcohaszing]

I started working on two big features recently: MDX 2 support in Prettier and MDX intellisense. It turns out this is a bit much, so I think I can better focus on one feature at a time, which is intellisense. If @j-f1 wants to continue working on #13676, I definitely encourage that.

These are the steps that are needed to fix MDX 2 support:

• Ignore MDX at first, just make sure the markdown tests work.
• Update to the latest version of the remark-* libraries and unified. Just updating already fixes some bugs in the snapshot tests, but it also introduces new ones. Also some formatting is changed, which isn’t more or less correct.

If all markdown tests pass, start considering MDX syntax. Any custom MDX parsing should be replaced with remark-mdx. This will introduce the following new node types that need to be handled:

• mdxFlowExpression ({2 * Math.PI} / {/* comment */}): This node contains a JavaScript expression that is already parsed to estree using acorn, so Prettier already knows how to handle its contents.
• mdxJsxFlowElement (<img src={props.avatar} />): This node looks like JSX, but it’s different, because its children are MDX. Its attributes are represented using estree.
• mdxjsEsm (import 'foo', export function bar() {}): These nodes contain import and export declarations parsed to estree. Formatting these should be straight-forward.

[j-f1]

If you’re able, can you push what you have for MDX 2 so far to your GitHub? I didn’t really start looking into fixing bugs, so if you did even a bit of that I would love to bring your changes into my PR (with attribution of course)