Error occur when build with next-mdx-remote and next.js standalone option

[EiffelFly]

Context

Hi!! Thank you for building and maintaining such a great package. I am encountering some issues when using codehike + next-mdx-remote + Nextjs with the standalone option.

During development, the code goes smoothly. But after building it with the Next.js standalone option, the app can't find some file that codehike relied on and throw Internal Error.

error running remarkCodeHike [Error: ENOENT: no such file or directory, open '/app/node_modules/.pnpm/[email protected]/node_modules/shiki/languages/abap.tmLanguage.json'] {
code-hike-test-2 |   errno: -2,
code-hike-test-2 |   code: 'ENOENT',
code-hike-test-2 |   syscall: 'open',
code-hike-test-2 |   path: '/app/node_modules/.pnpm/[email protected]/node_modules/shiki/languages/abap.tmLanguage.json'
code-hike-test-2 | }

The way to reproduce

1. Please clone this repo https://github.com/EiffelFly/code-hike-remote-docker
2. install everything and use the script npm run build
3. Activate the standalone server with node ./.next/standalone/server.js
4. goto http://localhost:3000

I have set up a docker image if this could help.

1. npm run docker:app:develop
2. goto http://localhost:3000

[thecuvii]

same with MDX-Bundler.

RequestId: eeabce19-9e2c-4811-a253-f8e2bb47d004 Error: Runtime exited with error: exit status 1
Runtime.ExitError
2022-10-02T15:50:03.592Z	ceffc43d-627d-4def-b9ef-37d29e9be86d	ERROR	error running remarkCodeHike [Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/[email protected]/node_modules/shiki/languages/abap.tmLanguage.json'] {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/var/task/node_modules/.pnpm/[email protected]/node_modules/shiki/languages/abap.tmLanguage.json'
}
✘ [ERROR] [plugin @mdx-js/esbuild] Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/[email protected]/node_modules/shiki/languages/abap.tmLanguage.json'
_mdx_bundler_entry_point-285f59bf-4135-4dab-93ee-7e00da3e41fc.mdx:0:0:
0 │ # GFM
╵ ^
2022-10-02T15:50:03.651Z	ceffc43d-627d-4def-b9ef-37d29e9be86d	ERROR	Error: Build failed with 1 error:
_mdx_bundler_entry_point-285f59bf-4135-4dab-93ee-7e00da3e41fc.mdx:0:0: ERROR: [plugin: @mdx-js/esbuild] Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/[email protected]/node_modules/shiki/languages/abap.tmLanguage.json'
    at failureErrorWithLog (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1566:15)
    at /var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1024:28
    at runOnEndCallbacks (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1438:61)
    at buildResponseToResult (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1022:7)
    at /var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1134:14
    at responseCallbacks.<computed> (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:671:9)
    at handleIncomingPacket (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:726:9)
    at Socket.readFromStdout (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:647:7)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12) {
  errors: [
    {
      detail: [Error],
      id: '',
      location: [Object],
      notes: [],
      pluginName: '@mdx-js/esbuild',
      text: "Error: ENOENT: no such file or directory, open '/var/task/node_modules/.pnpm/[email protected]/node_modules/shiki/languages/abap.tmLanguage.json'"
    }
  ],
  warnings: [],
  page: '/content/[category]/[name]/bundler'
}
RequestId: ceffc43d-627d-4def-b9ef-37d29e9be86d Error: Runtime exited with error: exit status 1
Runtime.ExitError

[pomber]

Both are using pnpm? I'll try to test it tomorrow

[thecuvii]

Ok, that's a problem with shiki and next.js.

Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build.

You can check out by build next.js with standalone, then go to .next/standalone/node_modules.

const nextConfig = {
    output: 'standalone'
}

The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.

// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};

After add above lines, CodeHike with next-remote-mdx works fine for me .

[thecuvii]

Ok, that's a problem with shiki and next.js.

Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build.

You can check out by build next.js with standalone, then go to .next/standalone/node_modules.

const nextConfig = {
    output: 'standalone'
}

The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.

// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};

After add above lines, CodeHike with next-remote-mdx works fine for me .

reference:

https://stackoverflow.com/questions/71422446/nextjs-force-dependency-with-outputstandalone-option

https://github.com/shikijs/shiki/issues/138#issuecomment-1057471160

[thecuvii]

Above solution not works for mdx-bundler.

Basically same error with https://github.com/kentcdodds/mdx-bundler/issues/158

[EiffelFly]

Both are using pnpm? I'll try to test it tomorrow

Yes i am using pnpm!

[EiffelFly]

Ok, that's a problem with shiki and next.js. Shiki import languages and themes by fs, when run next build, both languages and themes are tree-shaken by next.js (not if you manually import from your code), so not included in production build. You can check out by build next.js with standalone, then go to .next/standalone/node_modules.

const nextConfig = {
    output: 'standalone'
}

The solution is unstable_includeFiles, see: https://nextjs.org/docs/advanced-features/output-file-tracing#caveats.

// in your next.js page 
export const config = {
  // pnpm in my case, maybe 'node_modules/**/shiki/**/*.json' for npm or yarn.
  unstable_includeFiles: ['node_modules/.pnpm/**/shiki/**/*.json'],
};

After add above lines, CodeHike with next-remote-mdx works fine for me .

reference:

https://stackoverflow.com/questions/71422446/nextjs-force-dependency-with-outputstandalone-option

shikijs/shiki#138 (comment)

This is very interesting, thanks for your help! Let me test it further

[pomber]

Above solution not works for mdx-bundler.

Basically same error with kentcdodds/mdx-bundler#158

@Xwil what's the error with mdx-bundler? it may be a different issue

[thecuvii]

kentcdodds/mdx-bundler#158

✘ [ERROR] Could not resolve "@code-hike/mdx/dist/components.cjs.js"
_mdx_bundler_entry_point-228ef4a4-2912-45e8-b923-adb366b11779.mdx:3:109:
3 │ ...eSlot, Scrollycoding} from "@code-hike/mdx/dist/components.cjs.js";
╵                               ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can mark the path "@code-hike/mdx/dist/components.cjs.js" as external to exclude it from the bundle, which will remove this error.
2022-10-08T12:44:21.409Z	6601f87c-e5b2-4983-8777-41a6e196398a	ERROR	Error: Build failed with 1 error:
_mdx_bundler_entry_point-228ef4a4-2912-45e8-b923-adb366b11779.mdx:3:109: ERROR: Could not resolve "@code-hike/mdx/dist/components.cjs.js"
    at failureErrorWithLog (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1566:15)
    at /var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1024:28
    at runOnEndCallbacks (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1438:61)
    at buildResponseToResult (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1022:7)
    at /var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:1134:14
    at responseCallbacks.<computed> (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:671:9)
    at handleIncomingPacket (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:726:9)
    at Socket.readFromStdout (/var/task/node_modules/.pnpm/[email protected]/node_modules/esbuild/lib/main.js:647:7)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12) {
  errors: [
    {
      detail: undefined,
      id: '',
      location: [Object],
      notes: [Array],
      pluginName: '',
      text: 'Could not resolve "@code-hike/mdx/dist/components.cjs.js"'
    }
  ],
  warnings: [],
  page: '/content/[category]/[name]/bundler'
}
RequestId: 6601f87c-e5b2-4983-8777-41a6e196398a Error: Runtime exited with error: exit status 1
Runtime.ExitError

Local-dev works great, the error above was came from next.js server rendering.

[pomber]

you are using mdx-bundler inside getServerSideProps? why not getStaticProps?

[thecuvii]

you are using mdx-bundler inside getServerSideProps? why not getStaticProps?

Yeah, all pages are rendered inside getStaticProps, but not all of them are rendered at build stage.

https://nextjs.org/docs/api-reference/data-fetching/get-static-paths

[manduks]

this fixed the issue for me on getServerSideProps

// in your next.js page 
export const config = {
  unstable_includeFiles: ['node_modules/**/shiki/**/*.json'],
};

[MildTomato]

Also just hit this problem. Only found the issue by looking at logs in vercel 😂

// bottom of /pages/references/javascript/[...slug].tsx
// yes.. tried them all

export const config = {
  unstable_includeFiles: [
    'node_modules/.pnpm/**/shiki/**/*.json',
    'node_modules/**/shiki/**/*.json',
    'node_modules/.npm/shiki/**/*.json',
    '/var/task/node_modules/shiki/languages/abap.tmLanguage.json',
    'node_modules/shiki/languages/abap.tmLanguage.json',
    'node_modules/shiki/**/*.json',
  ],
}

[pomber]

Ideally the syntax highlighting should be done at build time, but I'm going to replace shiki with something custom soon and these problems should disappear.

[pomber]

I'm working on PR #308. There's a canary version there if anyone wants to try it and see if it fixes the problem.

[pomber]

should be fixed with v0.8.0, let me know how it goes

[mrboen94]

should be fixed with v0.8.0, let me know how it goes

Running v0.8.0, still running into this issue, Ill figure it out and report back though.

[pomber]

should be fixed with v0.8.0, let me know how it goes

Running v0.8.0, still running into this issue, Ill figure it out and report back though.

What's the error?

[mrboen94]

10:33:18 PM: ./node_modules/@code-hike/lighter/dist/index.esm.mjs
10:33:18 PM: Module not found: Can't resolve 'fs' in '/opt/build/repo/node_modules/@code-hike/lighter/dist'
10:33:18 PM: Import trace for requested module:
10:33:18 PM: ./node_modules/@code-hike/lighter/dist/index.esm.mjs
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: Module not found: Can't resolve 'fs' in '/opt/build/repo/node_modules/@code-hike/mdx/dist'
10:33:18 PM: Import trace for requested module:
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: > Build failed because of webpack errors```

[pomber]

10:33:18 PM: ./node_modules/@code-hike/lighter/dist/index.esm.mjs
10:33:18 PM: Module not found: Can't resolve 'fs' in '/opt/build/repo/node_modules/@code-hike/lighter/dist'
10:33:18 PM: Import trace for requested module:
10:33:18 PM: ./node_modules/@code-hike/lighter/dist/index.esm.mjs
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: Module not found: Can't resolve 'fs' in '/opt/build/repo/node_modules/@code-hike/mdx/dist'
10:33:18 PM: Import trace for requested module:
10:33:18 PM: ./node_modules/@code-hike/mdx/dist/index.browser.mjs
10:33:18 PM: > Build failed because of webpack errors```

That shouldn't happen, can you open a new issue with the steps to reproduce it?

[mrboen94]

I can do that, but I will need to find out how to reproduce it first, im trying to debug it on my end, then I'll open the issue.

EDIT: It's late so I will have to look into creating reproducible steps and issue tomorrow, in the meantime if you are using NextJs, add this to your next.config:

...
future: {
    webpack5: true,
},
webpack(config) {
    config.resolve.fallback = {
        ...config.resolve.fallback,
        fs: false,
    }
}
...