next.js icon indicating copy to clipboard operation
next.js copied to clipboard

sitemap in google search console says "Couldn't fetch"

Open Giorgi-Gamgebeli opened this issue 10 months ago • 52 comments

Discussed in https://github.com/vercel/next.js/discussions/75774

Originally posted by CodingReaper1 February 6, 2025

Link to the code that reproduces this issue

https://github.com/CodingReaper1/TheWildOasisAdminNextJs

To Reproduce

you don't need to reproduce I think

Current vs. Expected behavior

So I have created dynamic sitemap in nextjs which you can isee in my github repo in app folder problem is no matter what I can't successfully register sitemap on google search console.

I have seen discussion about this in github already https://github.com/vercel/next.js/issues/51649 however it said that I should create new issue.

I have tried everything shown in that github discussion

I went here www.xml-sitemaps.com/validate-xml-sitemap.html to test sitemap no issues there

I tried adding forward / or search queries in case request was cached

Image

I do have middleware I have page "/sitemap.xlm" in public routes then I tried completly disabling middleware on sitemap page but that didn't work either I have no idea what to do.

Provide environment information

operating system: windows 10
nextjs: 14
react: 18
you can see all packages in repo
live app if that helps somehow: the-wild-oasis-admin-nextjs.vercel.app

Which area(s) are affected? (Select all that apply)

Metadata

Which stage(s) are affected? (Select all that apply)

Vercel (Deployed)

Additional context

No response

Giorgi-Gamgebeli avatar Feb 09 '25 16:02 Giorgi-Gamgebeli

For how long have you had this issue? And, most importantly, can you reach the sitemap yourself? The Search Console can be a little annoying, because it is not live data, rather, it shows record of previous attempts IIRC. Over time, two, three days, it'll actually re-do them.

icyJoseph avatar Feb 10 '25 07:02 icyJoseph

@icyJoseph

I checked sitemap on this and it had no issue: www.xml-sitemaps.com/validate-xml-sitemap.html.

Sitemap is visible on url you can see it yourself: https://the-wild-oasis-admin-nextjs.vercel.app/sitemap.xml

And I have this issue about month right now at first I tried to verify only /sitemap.xml I waited and waited then try every possible way :D as you can see in image Searched google for solutions nothing found that works for me

Giorgi-Gamgebeli avatar Feb 10 '25 19:02 Giorgi-Gamgebeli

I mean, this is looking more like an issue with Search Console isn't it?

There's something called a Live test done from the search console, https://support.google.com/webmasters/answer/9012289?hl=en&sjid=17177608175401035119-EU#test_live_page, perhaps you can give that a go.

icyJoseph avatar Feb 10 '25 19:02 icyJoseph

@icyJoseph

Image

Image

Daily quota is bs btw I haven't done today anything or tried to index anything today for that matter to get this error There are no issues in manual actions, removals and security issues

ALSO I forgot to say only homepage is indexed nothing else but urls seems to have no problem

Also yea this might be issue with google search console not with nextjs I will try to get help there as well this is so stupid

Giorgi-Gamgebeli avatar Feb 11 '25 09:02 Giorgi-Gamgebeli

I has a similar issue, when using generateSitemaps @CodingReaper1

The difference is that I had sitemap.ts here: src/app/sitemap.ts. And because of using generateSitemaps() the sitemap was generated at website.com/sitemap/0.xml

Solution was to move sitemap.ts in an inner folder: That is: src/app/sitemaps/sitemap.ts

After this google is able to fetch from website.com/sitemaps/sitemap/0.xml

I think it's worth trying to move your sitemap.ts to ./sitemaps/sitemap.ts, and try using the new URL location (i.e https://the-wild-oasis-admin-nextjs.vercel.app/sitemaps/sitemap.xml

stefaneduard-deaconu avatar Feb 13 '25 08:02 stefaneduard-deaconu

Alright @stefaneduard-deaconu while I was trying what you told me I came across something that might fix this

I think problem all along was that sitemap read as html page

As said in https://github.com/vercel/next.js/issues/51649 I should have excluded sitemap completely from middleware matcher

Which I did but even testing with that gave error couldn't fetch

I didn't gave it time like week or two WHICH I think should be problem

So how I found out that page was read as html page

I did live test then I found this button which before I didn't even notice, I clicked it and on right it was showing html instead of sitemap right now it is showing sitemap instead of html because I excluded sitemap in middleware matcher Image

And in here it would show my root page "/" login page Image

My sitemap URL right now: https://the-wild-oasis-admin-nextjs.vercel.app/sitemap/sitemap.xml

My middleware matcher right now: Image

I also had sitemap at /sitemaps/sitemap.xml right now and moved it to /sitemap/sitemap.xml to see if it was caching problem because when I submited /sitemaps/sitemap.xml middleware was still running on sitemap page

I'm gonna wait week or two and see what's gonna happen right now it still says couldn't fetch Image

Weather it fixes it or not I will post results and if this doesn't fixes it Idk what will 🤣

Giorgi-Gamgebeli avatar Feb 13 '25 10:02 Giorgi-Gamgebeli

Facing the same issue for multiple sites

First Site

  • https://vatsal.website/sitemap.xml
  • https://github.com/nirnejak/vatsal-website

Second Site

  • https://rajarshigupta.com/sitemap.xml
  • https://github.com/nirnejak/rajarshi-website

nirnejak avatar Feb 13 '25 18:02 nirnejak

I have the same issue with my sitemap.xml how to solve the issue. i tried moving the sitemap.xml to a folder and tried but still same problem

Edit by maintainer bot: Comment was automatically minimized because it was considered unhelpful. (If you think this was by mistake, let us know). Please only comment if it adds context to the issue. If you want to express that you have the same problem, use the upvote 👍 on the issue description or subscribe to the issue for updates. Thanks!

JEEVADARSHAN avatar Feb 25 '25 15:02 JEEVADARSHAN

I completely forgot about this issue, because I started working on another project Nothing is changed on me I made some changes, but sitemap should be valid, but is not

Image

I actually uploaded another demo app which is part of the first application, which has same kind of issue? Image

I did fix the issue so I have no idea why is it still showing this 10 errors Image

Maybe it is actually uploaded but it doesn't show here because of cache?

Giorgi-Gamgebeli avatar Mar 23 '25 07:03 Giorgi-Gamgebeli

Seconding the solution from @stefaneduard-deaconu

I had the same error while trying to add sitemap.xml onto Google search console.

My hosting is on vercel and app is on nextjs while sitemap generation happens by fetching all blog post slug in a route.ts file.

I did 2 things:

  1. Moved the file to a subroute from /sitemap.xml/route.ts to /sitemaps/sitemap.xml/route.ts
  2. Excluded this route in my middleware.ts

Madhavkabra avatar Mar 29 '25 07:03 Madhavkabra

I switched to sitemap.ts approach from next js but the issue is still there

nirnejak avatar Mar 29 '25 09:03 nirnejak

I found a solution. Just add a slash at the end.

Just update

https:///www.example.com/sitemap.xml

to

https:///www.example.com/sitemap.xml/

Hope this can help.

polly3d avatar Apr 17 '25 02:04 polly3d

Hey there, is there any update regarding this problem? I am also facing this problem while the static sitemap from the public folder fetched in 1 minute but the dynamic one is not fetching.

nitin1kum avatar Jun 01 '25 16:06 nitin1kum

I completely forgot about this issue, because I started working on another project Nothing is changed on me I made some changes, but sitemap should be valid, but is not

Image

I actually uploaded another demo app which is part of the first application, which has same kind of issue? Image

I did fix the issue so I have no idea why is it still showing this 10 errors Image

Maybe it is actually uploaded but it doesn't show here because of cache?

Is your problem solved?

nitin1kum avatar Jun 01 '25 16:06 nitin1kum

I found a solution. Just add a slash at the end.

Just update

https:///www.example.com/sitemap.xml

to

https:///www.example.com/sitemap.xml/

Hope this can help.

This solved for me!

spennyp avatar Jun 09 '25 20:06 spennyp

I found a solution. Just add a slash at the end.

Just update

https:///www.example.com/sitemap.xml

to

https:///www.example.com/sitemap.xml/

Hope this can help.

Where did u add / if u mean google search console it doesn't work even after all this time it still says couldn't fetch This is beyond stupid at this point

Giorgi-Gamgebeli avatar Jun 10 '25 09:06 Giorgi-Gamgebeli

I found solution that worked for me! finally...

  1. Remove the sitemap from search console.
  2. In vercel go to Project Settings -> Data Cache -> Purge Cache -> Purge Everything.
  3. Add the sitemap to search console.

orjerby avatar Jun 12 '25 12:06 orjerby

I found solution that worked for me! finally...

  1. Remove the sitemap from search console.
  2. In vercel go to Project Settings -> Data Cache -> Purge Cache -> Purge Everything.
  3. Add the sitemap to search console.

not working, can you share your sitemap?

hussain992 avatar Jun 17 '25 04:06 hussain992

I found solution that worked for me! finally...

  1. Remove the sitemap from search console.
  2. In vercel go to Project Settings -> Data Cache -> Purge Cache -> Purge Everything.
  3. Add the sitemap to search console.

not working, can you share your sitemap?

Sure, here it is: https://blog-jet-iota.vercel.app/sitemap.xml

orjerby avatar Jun 17 '25 10:06 orjerby

  1. In vercel go to Project Settings -> Data Cache -> Purge Cache -> Purge Everything.

This likely wouldn't have been the "fix". GSC and Googlebot maintain their own independent cache, separate from Vercel’s cache, so purging Vercel's cache wouldn't force GSC to fetch a new version from the server.

A better method would be renaming the sitemap file (e.g., sitemap.xml to sitemap2.xml) as GSC will treat this as a completely new resource, bypassing any cached version and forcing Googlebot to fetch the updated sitemap immediately upon submission.

zevnda avatar Jun 17 '25 22:06 zevnda

  1. In vercel go to Project Settings -> Data Cache -> Purge Cache -> Purge Everything.

This likely wouldn't have been the "fix". GSC and Googlebot maintain their own independent cache, separate from Vercel’s cache, so purging Vercel's cache wouldn't force GSC to fetch a new version from the server.

A better method would be renaming the sitemap file (e.g., sitemap.xml to sitemap2.xml) as GSC will treat this as a completely new resource, bypassing any cached version and forcing Googlebot to fetch the updated sitemap immediately upon submission.

Interesting... I tried every single suggested solution, including changing the sitemap file name, but nothing worked.

orjerby avatar Jun 17 '25 22:06 orjerby

The issue is that Google search console expects a xml file with utf-8 encoded and also cors should not block the request, so just head over to next.config.js and add these headers

async headers() { return [ { source: '/sitemap.xml', headers: [ { key: 'Content-Type', value: 'text/xml; charset=utf-8', }, { key: 'Access-Control-Allow-Origin', value: '*' }, { key: 'Access-Control-Allow-Methods', value: '*' }, { key: 'Access-Control-Allow-Headers', value: '*' } ], }, ]; }

and this should work fine

Arun-ez avatar Jun 20 '25 19:06 Arun-ez

https://calcly.xyz/sitemap.xml?force=1

Adding a parameter force=1 solved my problem.

Joffey avatar Jun 25 '25 08:06 Joffey

I attempted all the "fixes" suggested in this issue and many others, but none resolved the problem. For context, I use next-sitemap to generate sitemaps and robots.txt at build time.

I noticed @Joffey say they had success, but also noticed they use a custom domain, so I figured I would try that, and it worked.

So the only solution that worked for me was adding my own custom domain to the deployment and setting up redirects, like so:

https://steamgameidler.vercel.app → 301 redirect → https://steamgameidler.com
https://www.steamgameidler.com → 301 redirect → https://steamgameidler.com
https://steamgameidler.com → Production

After that, I added the custom domain as a property in GSC and successfully submitted the full domain and sitemap route without issues.

When using a vercel.app domain, GSC requires selecting "URL Prefix" instead of "Domain" as the property type. This may limit sitemap submissions because the root domain (vercel.app) is predefined. I don't know how or why this would be an issue, but it's my best guess, because other than the redirects, the only differences are:

  • With a vercel.app property, GSC allows only the sitemap route (e.g., /sitemap.xml):
    Image

  • With a custom domain property, you can submit the full domain and route (e.g., https://steamgameidler.com/sitemap.xml):
    Image

I still cannot get the sitemap to work with the vercel.app property. Why? Not a clue. But based on these findings, it seems the issue might be with how vercel.app domains are handled in GSC.

zevnda avatar Jun 25 '25 12:06 zevnda

Did anyone find a fix yet? i'm having the same problem.

Edit by maintainer bot: Comment was automatically minimized because it was considered unhelpful. (If you think this was by mistake, let us know). Please only comment if it adds context to the issue. If you want to express that you have the same problem, use the upvote 👍 on the issue description or subscribe to the issue for updates. Thanks!

Krish209 avatar Jul 01 '25 20:07 Krish209

How such a core and essential feature doesn't work? I have tried everything that people suggested here and it still comes back as "Sitemap could not be read".

Image

What has been done:

  • updated middleware
  • made sure that page returns as "text/xml
  • tried adding parameters
  • custom domain connected
  • my google search console has custom domain property
Image

nicklaunches avatar Jul 15 '25 20:07 nicklaunches

Any update?

Edit by maintainer bot: Comment was automatically minimized because it was considered unhelpful. (If you think this was by mistake, let us know). Please only comment if it adds context to the issue. If you want to express that you have the same problem, use the upvote 👍 on the issue description or subscribe to the issue for updates. Thanks!

l-you avatar Jul 21 '25 15:07 l-you

i think (vercel.app) subdomains won't support by GSC. i observed that custom domains will get success for sitemap.xml

Sreenivas7463 avatar Jul 22 '25 05:07 Sreenivas7463

@Sreenivas7463 No, it does not work for any domain. Next.js 15.4.2. For example, https://revotale.com/sitemap.xml

l-you avatar Jul 22 '25 13:07 l-you

@l-you Means you are saying that Next.js 15.4.2 Sitemaps won't Support by Google Search Console ?

i am using Umami for analytics. https://sreeanalyticsumami-llli.vercel.app/share/MrAUKaTlGaWoVMns

i am observing this https://www.rybbit.io it is also Analytics tool Made by you rite.

Sreenivas7463 avatar Jul 23 '25 10:07 Sreenivas7463