Add sitemap so search engine results can showcase community and extensions, etc. just a bit better

[thadguidry]

When folks search "openrefine", many of the search engine results only show the following 4 areas:

DownloadDownload OpenRefine. OpenRefine is free software ...
User ManualThis manual is designed to comprehensively walk through ...
Running OpenRefineWith openrefine.exe​. You can run OpenRefine by double-clicking ...
Installing OpenRefineInstall or upgrade OpenRefine​ · The quick version: Install ...

And we'd ideally have a sitemap, so that way we can also easily showcase much more important (in my mind) /community (with the Discourse forum) or maybe only the forum? as well as /whats_new and /extensions. Maybe other things from the footer or top nav?

Here's the plugin: https://docusaurus.io/docs/api/plugins/@docusaurus/plugin-sitemap

[tfmorris]

Are you seeing pages which aren't being indexed? Sitemaps help with discoverability, but don't influence prioritization of results. Search engine priorities are driven by searchers, not publishers (and advertisers, of course, but that's a whole 'nother kettle of fish).

[thadguidry]

@tfmorris That is false information as of about circa ~ 2009, which you probably are not aware of, but that's fine. Here's some more information:

Include the URLs in your sitemap that you want to see in Google's search results. Google generally shows the canonical URLs in its search results, which you can influence with sitemaps. If you have different URLs for mobile and desktop versions of a page, we recommend pointing to only one version in a sitemap. However, if you want to point to both URLs, annotate your URLs to indicate the desktop and mobile versions.

• https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap

It is true, however, that for instance XML sitemaps, Google and others ignore certain things:

• Google ignores <priority> and <changefreq> values.
• Google uses the <lastmod> value if it's consistently and verifiably (for example by comparing to the last modification of the page) accurate.

[thadguidry]

I think I see what is going on. Here's what I'm seeing:

Not seeing Forum (instead I see Community being ranked higher). But I think that's because of it's subdomain perhaps? Still it's something I would more appreciate that if someone typed "openrefine help" that the Forum would have higher rank across many search engines, Google, Bing, Yandex, DuckDuckGo, etc. but I'm just not seeing that. Perhaps we need to improve the metadata on the Forum?

Not seeing Extensions unless I explicitly ask for them "openrefine extensions", but overall, I think instead the improvement we'd likely want to see for users would be directly on our Downloads page, with a section that says Extensions, and then links to the Extension page? That seems like a better way to advertise that there are extensions for OpenRefine, because I doubt folks would think about typing "openrefine extensions" as a new user, and much more likely to spot that we have extensions when the user is on the Download page, as long as Extensions are prominently displayed there?

[tfmorris]

@tfmorris That is false information as of about circa ~ 2009, which you probably are not aware of, but that's fine.

Srsly?

[mynk2611]

Hi @thadguidry,

I’ve been working on addressing this issue and tried increasing the priority of the specific pages (e.g., /community, /extensions, and /whats_new) in the sitemap. However, despite my efforts, the priority remained the same (0.5) across all pages.

While researching, I came across this discussion over SEO (https://github.com/facebook/docusaurus/issues/2604), which mentions that adding a lastmod field in the sitemap.xml could improve SEO. Currently, the sitemap doesn’t include a lastmod field.

Would it be worth considering adding the lastmod field to enhance SEO for these key pages? I’m happy to hear your thoughts or any recommendations on how best to proceed.

Looking forward to your input!

[thadguidry]

@mynk2611 you can add the lastmod field, it does help somewhat. The problem is that of @magdmartin or others maintaining the sitemap either manually or programmatically -because I'm not doing it ;-)

[wetneb]

I would hope that the sitemap is automatically generated by Docusaurus, I am really not keen to maintain it manually.

[mynk2611]

Hi @wetneb,

Thanks for your input! You’re correct—the sitemap is automatically generated. Here’s a screenshot for reference:

Currently, it doesn’t include the lastmod field. However, by adding a couple of plugins, it can be configured to include this field. Here’s an example of how the sitemap would look with the lastmod field:

Would you recommend moving forward with this change to enhance SEO? Looking forward to your thoughts!

[wetneb]

It depends how hacky the solution is, if it involves "a couple of plugins" I am perhaps vaguely worried that this is pretty complicated. I'd also be happy with waiting for Docusaurus to support this directly. It's probably easier to discuss this with a PR to look at.

[tfmorris]

I'm assuming that #406 was intended to close this.