ultimate-sitemap-parser icon indicating copy to clipboard operation
ultimate-sitemap-parser copied to clipboard

How to set timeout properly

Open ebauch opened this issue 3 years ago • 1 comments

Hi Team - thanks for this great lib! I have a couple of websites on my list that seem to be down. However, usp doesn't time out until like 2 hours later. How can I set the timeout to some shorter value for unresponsive websites? I did see the set_timeout parameter for the web_client class but not exactly sure how to initiate a web_client class with a proper timeout set.

Here is how far I have come:

from usp.web_client.requests_client import AbstractWebClient
from usp.web_client.requests_client import RequestsWebClient

RequestsWebClient.set_timeout(RequestsWebClient, timeout=10)
RequestsWebClient.set_max_response_data_length(RequestsWebClient, max_response_data_length=None)
web_client = RequestsWebClient

sitemap_tree_for_homepage('https://garfieldre2.org/', web_client=web_client)

which results in the following error.

image

ebauch avatar Feb 24 '22 02:02 ebauch

same issue here

fjfricke avatar Oct 20 '23 14:10 fjfricke