ultimate-sitemap-parser
ultimate-sitemap-parser copied to clipboard
How to set timeout properly
Hi Team - thanks for this great lib! I have a couple of websites on my list that seem to be down. However, usp doesn't time out until like 2 hours later. How can I set the timeout to some shorter value for unresponsive websites? I did see the set_timeout
parameter for the web_client
class but not exactly sure how to initiate a web_client class with a proper timeout set.
Here is how far I have come:
from usp.web_client.requests_client import AbstractWebClient
from usp.web_client.requests_client import RequestsWebClient
RequestsWebClient.set_timeout(RequestsWebClient, timeout=10)
RequestsWebClient.set_max_response_data_length(RequestsWebClient, max_response_data_length=None)
web_client = RequestsWebClient
sitemap_tree_for_homepage('https://garfieldre2.org/', web_client=web_client)
which results in the following error.
same issue here