Can't add sitemap under userAgent: '*'

[iloveip]

Hi there,

I have the following options for this plugin:

{
  resolve: 'gatsby-plugin-robots-txt',
  options: {
    host: siteUrl,
    sitemap: `${siteUrl}/sitemap-index.xml`,
    policy: [
      {
        userAgent: '*',
        sitemap: `${siteUrl}/sitemap-index.xml`
      },
      {
        userAgent: 'Yandex',
        host: siteUrl,
        sitemap: `${siteUrl}/sitemap-index.xml`
      }
    ]
  }
}

But in my output file I have:

User-agent: *

User-agent: Yandex
Sitemap: https://www.website.com/sitemap-index.xml
Host: https://www.website.com

How can I add sitemap under User-agent: *?

[FactoryAidan]

Your options{...} object is correct. But the objects in policies[{},...] do not support properties of host & sitemap; they are ignored there. And ignoring them there may be proper.

However there is a major issue here. The real issue is that in the resulting output, the Host & Sitemap properties are supposed to have a blank line of padding to show separation from directives like User-agent; they are not supposed to live inside of them.

Your question was how do you add Host & Sitemap definitions specific to each User-agent definition but the real question needs to be:

How do I get this plugin to put the proper newline padding above or below the global Host & Sitemap definitions so they aren't interpreted as part of the last User-agent definition written.

[iloveip]

Hi there,

I changed the options for this plugin in my gatsby-config.js to this:

{
  resolve: 'gatsby-plugin-robots-txt',
  options: {
    host: siteUrl,
    sitemap: `${siteUrl}/sitemap-index.xml`,
    policy: [{userAgent: '*'}]
  }
}

And in my output file now I have:

User-agent: *
Sitemap: https://www.website.com/sitemap-index.xml
Host: https://www.website.com

Is this correct or I still need to have a new line before Sitemap?