modrinth
Lack of sitemap means mods don't show up on Google
Environment
Website
Describe the bug
Unless they have been on the platform for a long time, mods do not show up when searched on Google. It seems like the dynamic nature of the website content means that Google's crawlers are unable to properly index all of the mods hosted on Modrinth automatically.
One way around this, especially for new sites and ones with dynamic content, is to produce a sitemap.xml for Google to collect all the relevant pages for it to index. Just to test, I went to modrinth.com/sitemap.xml and actually got an internal Nuxt error instead of a proper 404, which tells me there is something set up for it, but it isn't functioning right now.
If we could dynamically generate the sitemap.xml to include all the hosted mods (and modpacks) we could hopefully restore getting mods in Google search results again.
Just as a sidenote, sitemap.xml is apparently limited to 50k URLs and an uncompressed file size of 50MB, so it might be worthwhile to set up a tree of sitemaps, where the one at root lists additional sitemaps for all the subareas of the site (/users, /mods, etc), and even one step further to future proof it, the one at /mods could then point to "buckets" of mod URLs, i.e. mods1, mods2, and so on where each one contains up to 50k entries.
Steps To Reproduce
No response
Expected behavior
No response
Additional context
No response
I don't think they have a sitemap still (and neither does the rewrite apparently?), but regardless mods have been visible on search engines for awhile now, I think this can safely be closed?
