python-sitemap

python-sitemap copied to clipboard

Hi! Could you tell please how to exclude web pages with meta robots "noindex"

uninstalllexe

Alphabetize URLs

1

For large sitemaps, it's tough to see what changed when you rebuild it. That's because the URLs are not written in alphabetical order, so the diff doesn't work to see...

Garrett-R

Iframes are sometimes used to have parts of sites controlled by a CMS. Here we add the option of inspecting the iframe's content and for any links that are to...

marshvee

Web apps that use React, Vue.js, Svelte, etc. and client-side rendering don't work, because this package does not execute the JS, and so doesn't see all the content, thereby missing...

Garrett-R

Iframes are sometimes used to have parts of sites controlled by a CMS. It would nice to have the option of inspecting the iframe's content and for any links that...

Garrett-R

If I use the `--parserobots` argument the crawler simply crawls the main domain but doesn't follow any links dispite the allow directives. You can try it on `https://google.com` to see...

varna9000