Auto creating sitemap and index files from one large list?

[PTiCA1]

What problem does this feature solve?

Hello to everybody, does anyone have a solution to solve sitemap generation for a large number of pages? I generate articles from wordpress rest api and I need to split the sitemap into more files.

There is a solution in the sitemap.js documentation but I don't know how to use it in nuxt.js https://github.com/ekalinin/sitemap.js#building-just-the-sitemap-index-file

Thanks for any advice.

_{This feature request is available on Nuxt community (#c62)}

[AndyCosow]

Hi there, i was dealing with the similar problem - i made single sitemap with this module, there was little bit less than 50k links, so when provided that sitemap to google search console, they failed to fetch it.

My solution was to build sitemap index and put all the sub sitemaps in to it - i took all links from my old sitemap and divided them in categories. All this was done with sitemap.js in node.js using express router to send back response as generated sitemap.xml index file.

[roryheaney]

@AndyCosow did you have example code for that available?

[NicoPennec]

@PTiCA1 the "sitemap index" feature is WIP (see #6), but without limit size for now :confused:

After a quick review, it seems that we can't use the createSitemapIndex function from sitemap.js you're quoting, because the sitemap-module dynamically returns the sitemaps and won't be able to write to a folder as the feature proposes.

When the new version of the sitemap module with sitemapindex will be published (and stable ^^). We can start developing your feature. It will be an useful enhance 👍

[AndyCosow]

@roryheaney yes, here it is - On CodePile

I also recommend you to look in to sitemap.js documentation everything is pretty clear out there :)

[ricardogobbosouza]

@PTiCA1 This feature exists in v2.0.0 @NicoPennec can closed?

[NicoPennec]

@ricardogobbosouza nope! the v2.0.0 add only a manual splitting. The auto splitting is planned for the next release (see milestone). I started working on it (but not published yet)

[thinguyennam4059605]

Please release as soon as possible. Thanks

[thinguyennam4059605]

How to auto generate Image Sitemaps?

https://support.google.com/webmasters/answer/178636

[jd1378]

Hi @NicoPennec , sorry about mentioning, Any news on this ? how is it going so far ? I know everyone's busy , just wanted to know if It's around the corner or we need to do it ourselves for now Thanks

[nathanchase]

I would like to see this feature as well, as we have the need to have dynamic path names to split up a large list of urls, but the module doesn't await the response from the async function that generates the path names needed.

Essentially need it to wait for the following object, where the 1, 2, 3 is dynamic:

{
  {
    path: '/sitemap-products-1.xml'
    routes: [50kUrlsHere],
  },
  {
    path: '/sitemap-products-2.xml'
    routes: [50kUrlsHere],
  },
  {
    path: '/sitemap-products-3.xml'
    routes: [50kUrlsHere],
  },
  // etc...
}

[prinzt]

I managed to do this after some trial and error, via an async function returning the sitemap config. I stripped down the code a little bit, but basically that's how it works for me on a live system with nuxt-sitemap version 2.3.x

sitemapConfig.js // getData() is basically an axios wrapper

/**
 * Default Sitemap Options
 */
const defaultOptions = {
  lastmod: new Date(),
  exclude: ['/**'],
  changefreq: 'daily',
  priority: 1,
  gzip: true
}

const getSitemapChunks = async () => {
  const chunks = []
  const chunkSize = 40000
  // get the total number of items
  const { count } = await getData('/product-count')
  const chunkCount = Math.ceil(count / chunkSize)

  for (let i = 1; i <= chunkCount; i++) {
    chunks.push({
      ...defaultOptions,
      path: `sitemap-products-${i}.xml`,
      routes: async () => {
        const from = i * chunkSize - chunkSize
        const size = chunkSize - 1
        const products = await getData(
          `products?from=${from}&size=${size}`
        )
        const routes = []
        if (products) {
          products.forEach((product) => {
            routes.push(`/product/${product.slug}-${product.uid}`)
          })
        }
        return routes
      }
    })
  }
  return chunks
}

/**
 * Export global sitmapindex configuration as async function.
 * This enables splitting of large sitemaps
 *
 * @returns {object} Sitemapindex configuration object
 */
export const sitemap = async () => {
  const sitemaps = []

  /**
   * Create dynamic, chunked sitemaps
   */
  const sitemapChunks = await getSitemapChunks()
  sitemaps.push(...sitemapChunks)
  
  // you can add some more sitemaps ...

  
  return {
    changefreq: 'daily',
    priority: 1,
    lastmod: new Date(),
    cacheTime: 1000 * 60 * 60 * 12,
    exclude: ['/**'],
    path: '/sitemapindex.xml',
    sitemaps
  }
}

[nathanchase]

@prinzt How do you import ^^^^ (sitemapConfig.js) into your nuxt.config.js?

I tried your code, and it gave me a SyntaxError: Unexpected token 'export' error.

I also tried exporting it using module.exports = async () => - but even though that had no errors, it was if the sitemap-module ignored it completely and just generated a default sitemap.xml and sitemapindex.xml was 404.

EDIT: Never mind - we got it solved. module.exports = async () => did end up working after all. Thanks for sharing your solution!

[prinzt]

@nathanchase

the sitemapConfig.js is located in a custom folder "/configs/sitemapConfig.js" at the root level of my app.

in nuxt.config.js i do something like

//Simplified config:

import {  sitemap } from './configs/sitemapConfig'

export default {

  modules: [
    ['@nuxtjs/sitemap', sitemap],
  ]
}

The code in my first comment is also not my actual production code (there are some more things going on, that i stripped out), but somehow the essence of how it works. maybe there are some typos, but the logic should be clear.

[tol64]

Hello, everyone!

I ran tests with the code that @prinzt provided and found that when you change the number of articles in the blog, the sitemapindex.xml file does not change according to the number of articles.

I set the cacheTime parameter to 1 second to quickly track changes.

const getPostsSitemapChunks = async () => {
    let chunks = [];
    let chunkSize = 50;
    let response = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts-count`);
    let chunkCount = Math.ceil(response.data / chunkSize);

    for (let i = 0; i < chunkCount; i++) {
        chunks.push({
            path: `sitemap-posts${chunkCount < 2? '' : '-' + (i + 1)}.xml`,
            cacheTime: 1000,
            routes: async () => {
                const from = i * chunkSize;
                const size = chunkSize;

                let { data } = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts?from=${from}&size=${size}`);
                return data.map(post => {
                    return {
                        url: `/${post.language}/blog/post/${post.id}`,
                        changefreq: 'daily',
                        priority: 0.8,
                        lastmod: post.updated_at
                    }
                })
            }
        });
    }
    return chunks;
}

export const sitemap = async () => {
    let sitemaps = [];

    let postsSitemapChunks = await getPostsSitemapChunks();
    sitemaps.push(...postsSitemapChunks);

    return {
        path: '/sitemapindex.xml',
        cacheTime: 1000,
        lastmod: new Date(),
        sitemaps: sitemaps
    }
}

For example, if chunkSize is 50 and the server has 100 articles, then 2 files will be created during project compilation:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<sitemap>
<loc>http://localhost:3000/sitemap-posts-1.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

<sitemap>
<loc>http://localhost:3000/sitemap-posts-2.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

</sitemapindex>

Then create 200 articles in the database. But the file sitemapindex.xml remains the same as it was.

The sitemap files, which are connected to the sitemapindex.xml file, react to changes when the data changes on the server side, for example, when changing the sorting direction.

How do I make the sitemapindex.xml file also dynamic?

[barikhan1986]

Hello, everyone!

I ran tests with the code that @prinzt provided and found that when you change the number of articles in the blog, the sitemapindex.xml file does not change according to the number of articles.

I set the cacheTime parameter to 1 second to quickly track changes.

const getPostsSitemapChunks = async () => {
    let chunks = [];
    let chunkSize = 50;
    let response = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts-count`);
    let chunkCount = Math.ceil(response.data / chunkSize);

    for (let i = 0; i < chunkCount; i++) {
        chunks.push({
            path: `sitemap-posts${chunkCount < 2? '' : '-' + (i + 1)}.xml`,
            cacheTime: 1000,
            routes: async () => {
                const from = i * chunkSize;
                const size = chunkSize;

                let { data } = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts?from=${from}&size=${size}`);
                return data.map(post => {
                    return {
                        url: `/${post.language}/blog/post/${post.id}`,
                        changefreq: 'daily',
                        priority: 0.8,
                        lastmod: post.updated_at
                    }
                })
            }
        });
    }
    return chunks;
}

export const sitemap = async () => {
    let sitemaps = [];

    let postsSitemapChunks = await getPostsSitemapChunks();
    sitemaps.push(...postsSitemapChunks);

    return {
        path: '/sitemapindex.xml',
        cacheTime: 1000,
        lastmod: new Date(),
        sitemaps: sitemaps
    }
}

For example, if chunkSize is 50 and the server has 100 articles, then 2 files will be created during project compilation:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<sitemap>
<loc>http://localhost:3000/sitemap-posts-1.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

<sitemap>
<loc>http://localhost:3000/sitemap-posts-2.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

</sitemapindex>

Then create 200 articles in the database. But the file sitemapindex.xml remains the same as it was.

The sitemap files, which are connected to the sitemapindex.xml file, react to changes when the data changes on the server side, for example, when changing the sorting direction.

How do I make the sitemapindex.xml file also dynamic?

Any update about generate sitemapindex.xml dynamically without build site again?

[brandonpardede]

Hello, everyone!

I ran tests with the code that @prinzt provided and found that when you change the number of articles in the blog, the sitemapindex.xml file does not change according to the number of articles.

I set the cacheTime parameter to 1 second to quickly track changes.

const getPostsSitemapChunks = async () => {
    let chunks = [];
    let chunkSize = 50;
    let response = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts-count`);
    let chunkCount = Math.ceil(response.data / chunkSize);

    for (let i = 0; i < chunkCount; i++) {
        chunks.push({
            path: `sitemap-posts${chunkCount < 2? '' : '-' + (i + 1)}.xml`,
            cacheTime: 1000,
            routes: async () => {
                const from = i * chunkSize;
                const size = chunkSize;

                let { data } = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts?from=${from}&size=${size}`);
                return data.map(post => {
                    return {
                        url: `/${post.language}/blog/post/${post.id}`,
                        changefreq: 'daily',
                        priority: 0.8,
                        lastmod: post.updated_at
                    }
                })
            }
        });
    }
    return chunks;
}

export const sitemap = async () => {
    let sitemaps = [];

    let postsSitemapChunks = await getPostsSitemapChunks();
    sitemaps.push(...postsSitemapChunks);

    return {
        path: '/sitemapindex.xml',
        cacheTime: 1000,
        lastmod: new Date(),
        sitemaps: sitemaps
    }
}

For example, if chunkSize is 50 and the server has 100 articles, then 2 files will be created during project compilation:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<sitemap>
<loc>http://localhost:3000/sitemap-posts-1.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

<sitemap>
<loc>http://localhost:3000/sitemap-posts-2.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

</sitemapindex>

Then create 200 articles in the database. But the file sitemapindex.xml remains the same as it was.

The sitemap files, which are connected to the sitemapindex.xml file, react to changes when the data changes on the server side, for example, when changing the sorting direction.

How do I make the sitemapindex.xml file also dynamic?

Is there any update about this? Facing the same problem here

[duongthaison98]

Hello, everyone!

I ran tests with the code that @prinzt provided and found that when you change the number of articles in the blog, the sitemapindex.xml file does not change according to the number of articles.

I set the cacheTime parameter to 1 second to quickly track changes.

const getPostsSitemapChunks = async () => {
    let chunks = [];
    let chunkSize = 50;
    let response = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts-count`);
    let chunkCount = Math.ceil(response.data / chunkSize);

    for (let i = 0; i < chunkCount; i++) {
        chunks.push({
            path: `sitemap-posts${chunkCount < 2? '' : '-' + (i + 1)}.xml`,
            cacheTime: 1000,
            routes: async () => {
                const from = i * chunkSize;
                const size = chunkSize;

                let { data } = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts?from=${from}&size=${size}`);
                return data.map(post => {
                    return {
                        url: `/${post.language}/blog/post/${post.id}`,
                        changefreq: 'daily',
                        priority: 0.8,
                        lastmod: post.updated_at
                    }
                })
            }
        });
    }
    return chunks;
}

export const sitemap = async () => {
    let sitemaps = [];

    let postsSitemapChunks = await getPostsSitemapChunks();
    sitemaps.push(...postsSitemapChunks);

    return {
        path: '/sitemapindex.xml',
        cacheTime: 1000,
        lastmod: new Date(),
        sitemaps: sitemaps
    }
}

For example, if chunkSize is 50 and the server has 100 articles, then 2 files will be created during project compilation:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<sitemap>
<loc>http://localhost:3000/sitemap-posts-1.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

<sitemap>
<loc>http://localhost:3000/sitemap-posts-2.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

</sitemapindex>

Then create 200 articles in the database. But the file sitemapindex.xml remains the same as it was.

The sitemap files, which are connected to the sitemapindex.xml file, react to changes when the data changes on the server side, for example, when changing the sorting direction.

How do I make the sitemapindex.xml file also dynamic?

I have the same issue. I set the cacheTime to 1 and my sitemapindex.xml is updated only when i rebuild the project instead of F5

[ardeleandrei]

Hello, everyone!

I ran tests with the code that @prinzt provided and found that when you change the number of articles in the blog, the sitemapindex.xml file does not change according to the number of articles.

I set the cacheTime parameter to 1 second to quickly track changes.

const getPostsSitemapChunks = async () => {
    let chunks = [];
    let chunkSize = 50;
    let response = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts-count`);
    let chunkCount = Math.ceil(response.data / chunkSize);

    for (let i = 0; i < chunkCount; i++) {
        chunks.push({
            path: `sitemap-posts${chunkCount < 2? '' : '-' + (i + 1)}.xml`,
            cacheTime: 1000,
            routes: async () => {
                const from = i * chunkSize;
                const size = chunkSize;

                let { data } = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts?from=${from}&size=${size}`);
                return data.map(post => {
                    return {
                        url: `/${post.language}/blog/post/${post.id}`,
                        changefreq: 'daily',
                        priority: 0.8,
                        lastmod: post.updated_at
                    }
                })
            }
        });
    }
    return chunks;
}

export const sitemap = async () => {
    let sitemaps = [];

    let postsSitemapChunks = await getPostsSitemapChunks();
    sitemaps.push(...postsSitemapChunks);

    return {
        path: '/sitemapindex.xml',
        cacheTime: 1000,
        lastmod: new Date(),
        sitemaps: sitemaps
    }
}

For example, if chunkSize is 50 and the server has 100 articles, then 2 files will be created during project compilation:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<sitemap>
<loc>http://localhost:3000/sitemap-posts-1.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

<sitemap>
<loc>http://localhost:3000/sitemap-posts-2.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

</sitemapindex>

Then create 200 articles in the database. But the file sitemapindex.xml remains the same as it was.

The sitemap files, which are connected to the sitemapindex.xml file, react to changes when the data changes on the server side, for example, when changing the sorting direction.

How do I make the sitemapindex.xml file also dynamic?

Same issue. Can't update sitemaps at runtime with dynamic addresses. Ended up writing the sitemap.xml on the server with fs and updating it regurarily with a scheduled job.

[volk234]

Hello, everyone! I ran tests with the code that @prinzt provided and found that when you change the number of articles in the blog, the sitemapindex.xml file does not change according to the number of articles. I set the cacheTime parameter to 1 second to quickly track changes.

const getPostsSitemapChunks = async () => {
    let chunks = [];
    let chunkSize = 50;
    let response = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts-count`);
    let chunkCount = Math.ceil(response.data / chunkSize);

    for (let i = 0; i < chunkCount; i++) {
        chunks.push({
            path: `sitemap-posts${chunkCount < 2? '' : '-' + (i + 1)}.xml`,
            cacheTime: 1000,
            routes: async () => {
                const from = i * chunkSize;
                const size = chunkSize;

                let { data } = await axios.get(`${process.env.API_DOMAIN}/sitemap-posts?from=${from}&size=${size}`);
                return data.map(post => {
                    return {
                        url: `/${post.language}/blog/post/${post.id}`,
                        changefreq: 'daily',
                        priority: 0.8,
                        lastmod: post.updated_at
                    }
                })
            }
        });
    }
    return chunks;
}

export const sitemap = async () => {
    let sitemaps = [];

    let postsSitemapChunks = await getPostsSitemapChunks();
    sitemaps.push(...postsSitemapChunks);

    return {
        path: '/sitemapindex.xml',
        cacheTime: 1000,
        lastmod: new Date(),
        sitemaps: sitemaps
    }
}

For example, if chunkSize is 50 and the server has 100 articles, then 2 files will be created during project compilation:

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">

<sitemap>
<loc>http://localhost:3000/sitemap-posts-1.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

<sitemap>
<loc>http://localhost:3000/sitemap-posts-2.xml</loc>
<lastmod>2021-03-01T14:59:12.917Z</lastmod>
</sitemap>

</sitemapindex>

Then create 200 articles in the database. But the file sitemapindex.xml remains the same as it was. The sitemap files, which are connected to the sitemapindex.xml file, react to changes when the data changes on the server side, for example, when changing the sorting direction. How do I make the sitemapindex.xml file also dynamic?

Same issue. Can't update sitemaps at runtime with dynamic addresses. Ended up writing the sitemap.xml on the server with fs and updating it regurarily with a scheduled job.

Same trouble. Moved to self-generated sitemap because of this issue.

[FreekVR]

SImilar issues here --

When configuring a sitemap index with a sitemaps property using a Promise, this promsie runs during nuxt build and nuxt start, NOT when requesting the sitemap in browser.

This also has a nasty side-effect where if the endpoint from which we load our dynamic sitemap config crashes, the whole nuxt application also crashes and the whole site goes down. (we have a CMS which allows users to configure their sitemaps across multiple files)

Ideally, sitemap indexes sitemaps property would support promises in the same way as routes already works, where data is loaded only when requesting a sitemap via its URL.