ultimate-sitemap-parser icon indicating copy to clipboard operation
ultimate-sitemap-parser copied to clipboard

Published 20 hours ago • verified publisher icon mediacloud

Don't refetch sitemaps that were already fetched

Open pypt opened this issue 4 years ago • 0 comments 

2019-07-19 14:26:46,279 INFO mediawords.util.sitemap.media [95859/MainThread]: Fetching sitemap pages for media ID 10 (https://globalvoices.org/)...
2019-07-19 14:26:46,282 INFO usp.fetch_parse [95859/MainThread]: Fetching level 0 sitemap from https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,282 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,800 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/robots.txt...
2019-07-19 14:26:46,801 INFO usp.fetch_parse [95859/MainThread]: Fetching level 0 sitemap from https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,802 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,963 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap.xml...
2019-07-19 14:26:46,979 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:46,979 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:47,407 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:26:47,413 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:26:47,414 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:26:48,998 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
<...>
2019-07-19 14:54:06,695 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,696 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,869 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-home.xml...
2019-07-19 14:54:06,871 INFO usp.fetch_parse [95859/MainThread]: Fetching level 1 sitemap from https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:54:06,872 INFO usp.helpers [95859/MainThread]: Fetching URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
2019-07-19 14:54:07,033 INFO usp.fetch_parse [95859/MainThread]: Parsing sitemap from URL https://globalvoices.org/sitemap-posttype-post.201907.xml...
<...>

pypt avatar Jul 19 '19 11:07 pypt