sitemap-generator-cli icon indicating copy to clipboard operation
sitemap-generator-cli copied to clipboard

Published 20 hours ago • verified publisher icon lgraubner

Too much ignore pages

Open bshafiei-ir opened this issue 6 years ago • 5 comments

Hi I want to crawl my website but i get 128 ignore pages and just 4 added page. What is problem?

I ran this command: sitemap-generator -v -q https:\www.landa-sme.ir

result: Added 4 pages, ignored 128 pages, encountered 0 errors.

bshafiei-ir avatar Dec 01 '18 05:12 bshafiei-ir

I'm having the same issue got 100's of pages and it only adds 13 random ones. Not sure what the issue is as it's detecting the urls just not adding them.

dottodot avatar Dec 02 '18 08:12 dottodot

Interesting. I think it ignores them falsely because of connection issues.

@dottodot Can you provide an URL to check against?

lgraubner avatar Dec 02 '18 09:12 lgraubner

Yes this is the one I'm trying https://www.boxxerworld.com

dottodot avatar Dec 02 '18 09:12 dottodot

Interesting. I think it ignores them falsely because of connection issues.

@dottodot Can you provide an URL to check against?

And please check www.landa-sme.ir thanks

bshafiei-ir avatar Dec 02 '18 09:12 bshafiei-ir

For me it just queries the index.php page a bunch and then claims it's done

~/Downloads >>> sitemap-generator https://wiki.openstreetmap.org/wiki/Main_Page -f osmwiki.xml -v                                                                                                                             
(node:1158170) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
(Use `node --trace-warnings ...` to show where the warning was created)
[ ADD ] https://wiki.openstreetmap.org/wiki/Main_Page
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php
[ IGN ] https://wiki.openstreetmap.org/w/index.php

thibaultmol avatar Sep 05 '20 07:09 thibaultmol