Can not build more than 101 pages with Next.js

[kejiweixun]

What happened?

I clone markdoc's Next.js starter demo, randomly add 97 markdom files to pages/docs/ folder, then run npm run build, and Next.js will stuck on info - Creating an optimized production build ... forever.

To reproduce

1. git clone https://github.com/markdoc/markdoc-starter
2. Add random markdown files to pages/docs/ folder like below:

const fs = require("fs");
for (let i = 0; i < 98; i++) {
  fs.writeFileSync(`./pages/docs/${i}.md`, `##${i}`);
}

3. npm run build
4. It will stuck on info - Creating an optimized production build ... forever.
5. Delete one markdown file from pages/docs/, and it will build successfully.
6. And then, without deleting .next folder, add some more markdown files to pages/docs/, will build successfully. But if you delete .next before adding more files, it will hang forever.

I try to refactor my blog using markdoc, then found this issue. Had tried several next versions and markdoc versions, no luck. I think it's more likely markdoc related? Looking for help, thanks!

Version

try latest and older versions

Additional context

No response

[kejiweixun]

I think I know what's wrong now, but don't know how to fix it.

This issue is related to @markdoc/next.js. Somehow when having more than certain amount of markdown files, await this.importModule() just can't resolve, which cause the forever hang.

[faustbrian]

I think this is the exact same issue I'm encountering with https://github.com/vercel/next.js/issues/41453 where I have 400+ markdown files in total with 2 directories each containing 148 and the rest spread out through random directories. I started to delete markdown files one by one and eventually it would build.

[HonzaMac]

What does command 'limits' in terminal print?

[faustbrian]

You mean ulimit? Not aware of a command called limits.

[mfix-stripe]

Cc @ijjk in case you have ideas. I doubt this is a @markdoc/next.js bug, but it could be.

Regardless, I'm sure we could speed up our builds with some sort of caching.

[faustbrian]

Switching to MDX makes it possible to build my 400+ markdown files page. No chance to get it working with @markdoc/next.js even in the starter demo. So would think it's a bug with @markdoc/next.js or webpack? Though webpack seems less likely.

Edit 1 Based on logging in node_modules it never gets past https://github.com/markdoc/next.js/blob/569b557c540a8a3bb74b8e04cf91a97f51f90f0a/src/loader.js#L77 like @kejiweixun mentioned and it also never triggers the surrounding try/catch so the promise just endlessly resolves.

Edit 2 Note that await resolve(schemaDir, resource) actually resolves so it's await this.importModule that does not resolve.

Edit 3 Replacing await this.importModule with await import continues to try to actually build but fails for other reasons once it collected all files.

Edit 4 This issue seems to kick in as soon as you have a certain amount of markdown files AND additional nodes and tags in markdoc/tags.js and markdoc/nodes.js. Having a lot of markdown files and not using custom nodes and/or tags gets the site to built. Note that those don't have to be some buggy nodes or tags. You can copy ones from the documentation and it will still happen.

[mfix-stripe]

Hey folks, sorry for the delay here. I dug into this deeper, and there is something weird going on within Webpack and/or Next.js that prevents us from importing a module more than a certain amount of times at once. The trouble is we don't know that number ahead of time, nor are we able to stop if from hanging forever without introducing even worse behavior.

I fixed this issue here (https://github.com/markdoc/next.js/pull/19) by removing the build-time importModule call, which disables the Markdoc validation support we had previously included. If you want to validate all your content (for example in CI), you should call Markdoc.validate in a separate script.

I will release this fix on Monday.

Full validation was never supported in @markdoc/next.js so this shouldn't break any applications.

As a follow up, I will open an issue that re-thinks build time schema validation for the Markdoc Nexts.js plugin.