More sitemap types

[pedro-mendonca]

Hi,

I would like to suggest adding an optional checkbox to include an archive/category do sitemap.xml. The user could also choose some categories to include/exclude.

Regards Pedro Mendonça

[sybrew]

Hi Pedro,

The sitemap's going to be expanded through Extensions.

Free Extension: Categories, Tags, Images, News Sitemap, etc. ~Premium Extension: News sitemap, Image sitemap, etc.~

For now, the basic sitemap included in The SEO Framework is sufficient, or even perfect, for Search Engines.

Adding more onto that gives clutter and increases maintenance time and costs. I also do not want users to be lead into bad SEO practices ("It's in my sitemap, so it's OK...").

The sitemap, in this day and age, mustn't be considered the main factor for getting your pages found. Your overal website navigation (i.e. theme, post meta and menus) should indicate such existence.

The sitemap should only be used to notify Search Engines for what really matters: Post, public CPT and Page updates.

I hope this clears things up. I do want to expand the sitemap, but only justified for users that know why and how.

Cheers!

[pedro-mendonca]

Thank you for your feedback :+1:

[sybrew]

From ngocdang83's support topic:

Add ability to set sitemap locations.

[jkirker]

Yes, a filter to change the sitemap path would be great - or a settings option. It would make transitioning from Yoast easier as their sitemap url is different. Hate to have to go in and remodify all the various webmaster tools.

**It would also be awesome in the page/post/CPT SEO admin settings to have a checkbox to exclude a page from the sitemap rather than managing the list externally in functions. Just a thought.

[sybrew]

@jkirker You can remove pages with the in-post noindex option. Not having a page indexed is a very good reason to remove it from the sitemap as well and vice versa.

[bigstamp]

Was there any update with a news sitemap? as its crucial, if your listed in Google News

[sybrew]

Google explicitly states it doesn't use the News Sitemap as the only signal.

Google News doesn't favor sites that use News sitemaps in ranking results. Whether there's a News sitemap available or not, Google News uses normal crawl methods to search and index all news sites (homepage and sections).

In the wild, I've also seen sites pop up in the news-section just fine using the regular sitemap and with help of the Articles extension.

With this, I feel no obligation rushing this feature, and I am instead focussing on features that do have an impact: No ads. No distractions. No nonsense.

[bigstamp]

Yea I get your point, while it may not impact sites once they are indexed. But it does have an impact in the news showing up. With a site map my news articles are listed in google news within a few minute, without it’s a few hours. Dosnt seem much difference but for breaking news the difference is massive.

[sybrew]

Yea, that was in the back of my head too. This is the first time I've seen confirmation, however. I'll dig into it. Thanks!

[sybrew]

Because of the many misconceptions of the sitemap, and as some search engines (Bing, Yandex, Baidu) don't move fast enough, and because some features do give slight benefit in some circumstances, I'm going to transform how the sitemap works and is set up.

I'm going to move TSF to PHP 5.5 very soon. With the move of TSFEM from WordPress.org, I could finally gather useful information via the update server: 99% of the Extension Manager users are at PHP 5.6 or later, of which 60% are at PHP 7.1 or later. 1% is at PHP 5.5.

WordPress will enforce PHP 5.6 from April 2019, too.

I think my users are well aware of the technology they're using, and you are not here to be fooled around. I respect you for this.

The current sitemap's technology is ancient. I built it when I was still learning PHP. I added it in version 2.2.9, back in 2015.

---

Upcoming changes

The sitemap is going to:

0. Get a field*, wherein all "hooks" are prepared; so we can easily expand it.
1. Get an index.
2. Be split up by post type.
3. Get a PHP 5.5-style generator and iterator. This will take the strain off your server.
4. Be optionally listing categories and tags, at the bottom of each related type's sitemap.
5. Need a PR sent to WP Rocket that adheres to these adjustments.

*The field will likely consist of an abstract factory pattern, and something the "WordPress way," with filters and hooks. For example, this list will be filterable, whereafter the factory runs:

[
	'page' => [
		'xml' => {$built-in non-hierarchical factory}, // class name: Attained from get_supported_post_types() loop.
	],
	'post => [
		'xml' => {$built-in hierarchical factory}, // Attained from get_supported_post_types() loop.
	]
	'some_cpt' => [
		'xml' => {$built-in hierarchical factory}, // Example, attained from get_supported_post_types() loop.
	],
	'news' => [
		'xml'      => {$non-default factory},
		'xsl'      => {$non-default factory},
		'supports' => [ // maybe define this at factory level.
			'taxonomy' => false,
			'image'    => true,
		]
	],
	'video' => [
		'xml'      => {$non-default factory},
		'xsl'      => {$non-default factory},
		'supports' => [ // maybe define this at factory level.
			'taxonomy' => true,
		]
	],
]

ETA: v3.4 4.0.0.

See this comment for implementation: https://github.com/sybrew/the-seo-framework/issues/430#issuecomment-503608385 Also this repo for a working example: https://github.com/sybrew/tsf-term-sitemap

[sybrew]

From Kacper's suggestion:

Add pagination to the sitemap, imposing entry limits; so to reduce server load when the sitemap's building.

I've added this here as it should be considered in building the index map.

Kacper suggested 5000 posts per page. We'll test this, and we might want to transform the current post limit setting to this.

[sybrew]

My comment here explains how to integrate more sitemaps.

Via regular expressions (see first attached commit below), we can now integrate sitemap pagination.

[sybrew]

Here's a proof of concept sitemap generator for terms: https://github.com/sybrew/tsf-term-sitemap Requires TSF v4.0.0-rc1.5 or later.

[sybrew]

In https://github.com/sybrew/The-SEO-Framework-Extension-Manager/commit/faa6a7b048a7e140c14beb0d5f09166ac6fd993e, we added a Google News sitemap via the Articles extension. It also includes image markup.

Why Articles? Well, it has the capability to annotate posts as NewsArticles 😃

https://github.com/sybrew/the-seo-framework/issues/31#issuecomment-445472929 through https://github.com/sybrew/the-seo-framework/issues/31#issuecomment-447154704 are now marked as resolved.

[sybrew]

Punted to a later update due to limited resources (time).

[blizam]

The sitemap's going to be expanded through Extensions.

Free Extension: Categories, Tags, Images, News Sitemap, etc. ~Premium Extension: News sitemap, Image sitemap, etc.~

For now, the basic sitemap included in The SEO Framework is sufficient, or even perfect, for Search Engines.

I take it that Free Extension no longer exists?

If we want to include Category pages in the sitemap is this custom function the recommended solution: https://kb.theseoframework.com/kb/filter-reference-for-the-seo-framework/#add-categories-and-tags-to-base-sitemap

I think category pages, particularly Product Category pages in WooCommerce are important enough to either include by default or include/exclude via a checkbox in the base settings. I'm seeing several eCommerce sites with a high (not a majority... 10-30%) number of the category pages show up in Search Console as "Crawled - currently not indexed" and then Inspect URL, right at the top of it's report states: Discovery: Sitemaps - No referring sitemaps detected

[sybrew]

Hi @blizam,

You can fall back to the WordPress Core sitemaps by disabling the "optimized" sitemap. TSF augments the WordPress Core sitemaps for indexability and last-modified support.

The Google News sitemap has been implemented via Articles.

Images and Video sitemaps have been superseded by semantic HTML and structured data.

With that, I actually consider this issue resolved.

[blizam]

Thank you for the fast reply @sybrew -- if I wanted to retain the "indexability" (can you expand on that?) and last-modified support that TSF is adding, would the custom function I linked to initially still be the best way to go about including product categories?

I honestly had no idea WP had it's own sitemaps xml... more info here for customizing those for future travelers: https://make.wordpress.org/core/2020/07/22/new-xml-sitemaps-functionality-in-wordpress-5-5/

[sybrew]

Hi @blizam,

TSF determines "indexability" by testing if the page is indexable (not "noindex") and isn't redirected. You can use tsf()->sitemap()->utils()->is_post_included_in_sitemap() and is_term_included_in_sitemap() for that, see https://github.com/sybrew/the-seo-framework/blob/5.0.2/inc/classes/sitemap/utils.class.php#L72-L174.

The docs you linked to still need to be updated to reflect TSF v5.0's API -- I put it on the back burner, but I must get to it soon.

Still, I recommend sticking to what TSF already provides: It's easy, it works, and anything custom added to it gives little to no ROI. But, yes, the API example should still work, but it'll bombard your logs with deprecation notices. I'll update the documentation tomorrow.

[blizam]

Thank you :)

[sybrew]

Hi @blizam,

I just modernized the documentation to support TSF v5.0 and later. Here's the tested code you requested: https://tsf.fyi/kb/a/160#add-categories-and-tags-to-base-sitemap.

[blizam]

Awesome. Thank You!