sitemapper
sitemapper copied to clipboard
Add an option search in nested SiteMaps
Hi,
It would be nice to have an option to search in nested sitemaps like this: https://bigcrafters.com/sitemap.xml
Where we have an index map with:
- https://bigcrafters.com/sitemap-0.xml
- https://bigcrafters.com/server-sitemap-index.xml
- https://bigcrafters.com/server-sitemap-brands.xml
- https://bigcrafters.com/server-sitemap-categories.xml
- https://bigcrafters.com/server-sitemap-pages.xml
- https://bigcrafters.com/server-sitemap-products.xml
- https://bigcrafters.com/server-sitemap-sellers.xml
+1
Yes, just found this neat library, and unfortunately there is no way to get sub-sitemap info, e.g. if I want info about which sitemap(s) a url came from in my report. Here is code in sitemapper.js so it is definitely traversing the sub-sitemaps, but not giving us any info about them in the output (would be a really easy feature / option to add):
} else if (data && data.sitemapindex) {
// Handle child sitemaps found inside the active sitemap
if (this.debug) {
console.debug(`Additional sitemap found during "crawl('${url}')"`);
}
// Map each child url into a promise to create an array of promises
const sitemap = data.sitemapindex.sitemap.map(
(map) => map.loc && map.loc[0]
);