generateSitemaps is broken in production

[tomcru]

Link to the code that reproduces this issue

https://github.com/tomcru/nextjs-sitemap create next app with an /app/sitemap.ts file

import { MetadataRoute } from "next";

const PAGE_SIZE = 1000;

const fetchGames = async (pageNumber: number) => {
  const gamesResponse = await fetch(
    `https://api.gamegator.net/v1/products?pageSize=${PAGE_SIZE}&pageNumber=${pageNumber}&currency=usd&order=nameAsc`,
    {
      headers: {
        "content-type": "application/json",
      },
    }
  );

  return await gamesResponse.json();
};

export async function generateSitemaps() {
  return [
    { id: 1 },
    { id: 2 },
    { id: 3 },
    { id: 4 },
    { id: 5 },
    { id: 6 },
    { id: 7 },
    { id: 8 },
    { id: 9 },
    { id: 10 },
    { id: 11 },
    { id: 12 },
  ];
}

export default async function sitemap({
  id,
}: {
  id: number;
}): Promise<MetadataRoute.Sitemap> {
  console.log(id);
  const games = await fetchGames(id);

  return games.data.items.map((game: any) => ({
    url: game.slug,
    changeFrequency: "daily",
  }));
}

To Reproduce

npm run build, then npm run start

Current vs. Expected behavior

When running npm run build on Canary:

> [email protected] build
> next build

   ▲ Next.js 14.1.1-canary.50

   Creating an optimized production build ...
 ✓ Compiled successfully
 ✓ Linting and checking validity of types
 ✓ Collecting page data
 ✓ Generating static pages (17/17)
 ✓ Collecting build traces
 ✓ Finalizing page optimization

Route (app)                              Size     First Load JS
┌ ○ /                                    5.14 kB        90.1 kB
├ ○ /_not-found                          885 B          85.9 kB
└ ● /sitemap/[__metadata_id__]           0 B                0 B
    ├ /sitemap/1.xml
    ├ /sitemap/2.xml
    ├ /sitemap/3.xml
    └ [+9 more paths]
+ First Load JS shared by all            85 kB
  ├ chunks/54-e1835bee2a2e1d02.js        29.6 kB
  ├ chunks/fd9d1056-9bd6272302a380c1.js  53.5 kB
  └ other shared chunks (total)          1.86 kB


○  (Static)  prerendered as static content
●  (SSG)     prerendered as static HTML (uses getStaticProps)

Notice: console.log(id) (https://github.com/tomcru/nextjs-sitemap/blob/eaf5b7de57602e9a260457960e70ed1d983a938b/src/app/sitemap.ts#L40) missing in these logs.

Then running npm run start: /sitemap/1.xml returns not-found

When running npm run dev: /sitemap.xml/1 works

When running npm run build on next 14.1.0:

> [email protected] build
> next build

   ▲ Next.js 14.1.0

   Creating an optimized production build ...
 ✓ Compiled successfully
 ✓ Linting and checking validity of types
 ✓ Collecting page data
   Generating static pages (0/17)  [    ]6
7
5
4
   Generating static pages (0/17)  [=   ]8
1
9
1
1
1
3
   Generating static pages (10/17) [==  ]
 ✓ Generating static pages (17/17)
 ✓ Collecting build traces
 ✓ Finalizing page optimization

Route (app)                              Size     First Load JS
┌ ○ /                                    5.12 kB        89.3 kB
├ ○ /_not-found                          885 B            85 kB
└ ● /sitemap/[__metadata_id__]           0 B                0 B
    ├ /sitemap/1.xml
    ├ /sitemap/2.xml
    ├ /sitemap/3.xml
    └ [+9 more paths]
+ First Load JS shared by all            84.2 kB
  ├ chunks/69-1b6d135f94ac0e36.js        28.9 kB
  ├ chunks/fd9d1056-cc48c28d170fddc2.js  53.4 kB
  └ other shared chunks (total)          1.86 kB


○  (Static)  prerendered as static content
●  (SSG)     prerendered as static HTML (uses getStaticProps)

The console.log(id) (https://github.com/tomcru/nextjs-sitemap/blob/eaf5b7de57602e9a260457960e70ed1d983a938b/src/app/sitemap.ts#L40) works. Notice how it uses duplicate id values although generating different sitemaps!! ⚠️

When running npm run start, this creates working pages on: /sitemap/1.xml, but because of the duplicate ids: /sitemap/5.xml (for example) has the same contents as /sitemap/1.xml

I was initially generating the ids in generateSitemaps() from the API response, but noticed that even returning hard-coded ids (as in my reproduction) causes this issue.

I also noticed I was able to access any /sitemap/n.xml and it having some data.

Provide environment information

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 23.0.0: Fri Sep 15 14:42:57 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T8112
Binaries:
  Node: 20.5.0
  npm: 10.4.0
  Yarn: 1.22.19
  pnpm: 8.14.0
Relevant Packages:
  next: 14.1.1-canary.50 // Latest available version is detected (14.1.1-canary.50).
  eslint-config-next: 14.1.0
  react: 18.2.0
  react-dom: 18.2.0
  typescript: 5.3.3
Next.js Config:
  output: N/A

Which area(s) are affected? (Select all that apply)

App Router

Which stage(s) are affected? (Select all that apply)

next build (local)

Additional context

This issue seems to be related: https://github.com/vercel/next.js/issues/60894 +PR: https://github.com/vercel/next.js/pull/61088

_NEXT-2501

[tomcru]

@huozhi this might be of interest to you, since you were working on https://github.com/vercel/next.js/pull/61088

[tomcru]

This even happens in this minimal minimal example:

import { MetadataRoute } from "next";

export async function generateSitemaps() {
  return [
    { id: 1 },
    { id: 2 },
    { id: 3 },
    { id: 4 },
    { id: 5 },
    { id: 6 },
    { id: 7 },
    { id: 8 },
    { id: 9 },
    { id: 10 },
    { id: 11 },
    { id: 12 },
  ];
}

export default async function sitemap({
  id,
}: {
  id: number;
}): Promise<MetadataRoute.Sitemap> {
  console.log(id);

  return [
    {
      url: `test${id}`,
      changeFrequency: "daily",
    },
  ];
}

On build of Canary: /sitemap/1.xml is simply Not Found

On build of 14.1:

> [email protected] build
> next build

   ▲ Next.js 14.1.0

   Creating an optimized production build ...
 ✓ Compiled successfully
 ✓ Linting and checking validity of types
 ✓ Collecting page data
   Generating static pages (0/17)  [    ]5
4
3
6
7
9
8
1
1
1
   Generating static pages (10/17) [=   ]
1
 ✓ Generating static pages (17/17)
 ✓ Collecting build traces
 ✓ Finalizing page optimization

Route (app)                              Size     First Load JS
┌ ○ /                                    5.12 kB        89.3 kB
├ ○ /_not-found                          885 B            85 kB
└ ● /sitemap/[__metadata_id__]           0 B                0 B
    ├ /sitemap/1.xml
    ├ /sitemap/2.xml
    ├ /sitemap/3.xml
    └ [+9 more paths]
+ First Load JS shared by all            84.2 kB
  ├ chunks/69-1b6d135f94ac0e36.js        28.9 kB
  ├ chunks/fd9d1056-cc48c28d170fddc2.js  53.4 kB
  └ other shared chunks (total)          1.86 kB


○  (Static)  prerendered as static content
●  (SSG)     prerendered as static HTML (uses getStaticProps)

[meh7an]

For me this happens as well. It is fine until 9, but anything >10 is returning 1. Also, only digits work correctly. Therefore, anything except [1-9] as single character will not return as expected on production. It's while the dev platform works properly.

[tomcru]

@meh7an if you are looking for a solution in the meantime (not statically generated).

You can create an /app/sitemaps/products/[id]/route.ts (for example) endpoint and create your sitemaps there. Here's an example:

import { headers } from 'next/headers';
import { buildAbsoluteProductURL } from '../../../../common/utils/urls';
import { fetchProducts } from '../../utils/fetchProducts';
import { generateSitemapItem } from '../../utils/generateSitemap';

function getPageNumberFromURL(): string {
  const xUrl = headers().get('x-url');
  if (!xUrl) {
    throw new Error('x-url header is missing');
  }
  const currentPath = new URL(xUrl).pathname;
  const currentPage = currentPath.split('/').pop()?.split('.')[0];
  if (!currentPage) {
    throw new Error('Page number is missing or invalid');
  }
  return currentPage;
}

export async function GET() {
  try {
    const pageNumber = getPageNumberFromURL();

    const { data } = await fetchProducts(pageNumber);

    if (!data.items.length) {
      throw new Error('No items found for the given page number');
    }

    const sitemapXML = `<?xml version="1.0" encoding="UTF-8"?>
    <urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${data.items
          .map((item) => {
            const priority =
              Math.round((1 - (item.rank || 0) / data.totalCount) * 100) / 100;

            return generateSitemapItem(
              buildAbsoluteProductURL(item.slug),
              'daily',
              priority,
            );
          })
          .join('')} 
    </urlset>`;

    return new Response(sitemapXML, {
      headers: { 'Content-Type': 'text/xml' },
    });
  } catch (error) {
    console.error(error);
    return new Response('Internal Server Error', { status: 500 });
  }
}

The sitemap is then accessible via /sitemaps/products/1.xml, etc.

We also use a middleware passing x-url to the header to get access to the URL on the server.

[meh7an]

Thank you for the workaround. This is also helpful for using sitemapindex instead of urlset, since I find no way of auto-generating sitemapindex as well.

[tomcru]

Yep, also using this for the index like /app/sitemaps/index.xml/route.ts, which then ends up looking like:

import { URLS } from '../../../common/configs/urls';
import { fetchProducts } from '../utils/fetchProducts';
import { fetchUsers } from '../utils/fetchUsers';
import { generateSitemapLink } from '../utils/generateSitemap';

const generateSitemapLink = (url: string) =>
  `<sitemap><loc>${url}</loc></sitemap>`;

export async function GET() {
  const { data: productsData } = await fetchProducts('1');
  const { data: usersData } = await fetchUsers('1');

  const sitemapIndexXML = `<?xml version="1.0" encoding="UTF-8"?>
    <sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
        ${generateSitemapLink(`${URLS.base}/sitemaps/sitemap.xml`)}

        ${Array.from({ length: productsData.pageCount }, (_, i) => i + 1)
          .map((id) =>
            generateSitemapLink(`${URLS.base}/sitemaps/games/${id}.xml`),
          )
          .join('')} 

        ${Array.from({ length: usersData.pageCount }, (_, i) => i + 1)
          .map((id) =>
            generateSitemapLink(`${URLS.base}/sitemaps/users/${id}.xml`),
          )
          .join('')}
    </sitemapindex>`;

  return new Response(sitemapIndexXML, {
    headers: { 'Content-Type': 'text/xml' },
  });
}

[meh7an]

So here's how I managed to implement my sitemap system:

/sitemap.xml-> Static xml showing all sitemapindexes /sitemap_static.xml -> Static webpages linked as urlset /products/sitemap.js -> dynamically indexing products using built-in functions /categories/sitemap.js -> same as products /api/sitemaps/products.xml/route.js -> listing all product sitemaps in a sitemapindex /api/sitemaps/categories.xml/route.js -> same as above

I increased products per page, so the count doesn't increase more than 9 to cause the bug for now, until a proper fix.

I assume implementing sitemapindex, and id naming with no limitations as a fix for this issue.

[hc0503]

Hi @huozhi , Please check this issue. #60894 The issue was fixed in canary version, but now released v14.1.1 version has the error again.

[huozhi]

This change is not included in 14.1.1 yet 🙏 please use canary to get benefits of the fix

[tomcru]

Can confirm this issue is fixed in Next.js 14.1.1-canary.82 when I just tried it with my reproduction. Thank you @huozhi & @abhinaypandey02 ❤️

[huozhi]

This fix is landed in 14.1.2

[github-actions[bot]]

This closed issue has been automatically locked because it had no new activity for 2 weeks. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.