Add option to extend read timeout / solve `Read timed out. (read timeout=15)` error on plain-requests when the request needs more than 15 seconds

[adhawkins]

Version and OS v0.39.19.1 on Docker

Is your feature request related to a problem? Please describe. I have a check that monitors a PHP module on PECL.

https://pecl.php.net/package/timezonedb

This check always fails with the following error:

HTTPSConnectionPool(host='pecl.php.net', port=443): Read timed out. (read timeout=15)

If I visit the site in a browser, it takes a noticeably long time to display. Using wget takes over 30 seconds to retrieve the content:

[andy@xcp-docker-amd ~]$ time wget -O/dev/null https://pecl.php.net/package/timezonedb
--2022-09-27 09:32:06--  https://pecl.php.net/package/timezonedb
Resolving pecl.php.net (pecl.php.net)... 104.236.228.160
Connecting to pecl.php.net (pecl.php.net)|104.236.228.160|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘/dev/null’

/dev/null               [  <=>               ] 127.47K   403KB/s    in 0.3s

2022-09-27 09:32:38 (403 KB/s) - ‘/dev/null’ saved [130531]


real    0m31.802s
user    0m0.018s
sys     0m0.006s

Describe the solution you'd like Add an environment variable to extend the read timeout.

Describe the use-case and give concrete real-world examples This site takes a long time to display in a browser, and always fails in changedetection.io:

https://pecl.php.net/package/timezonedb

[dgtlmoon]

You're using plain http requests (not chrome) right?

[adhawkins]

You're using plain http requests (not chrome) right?

That's correct.

[k4deng]

+1 for this feature

[melalj]

Something might have broke since this ticket: https://github.com/dgtlmoon/changedetection.io/issues/369 It seems changedetection is no longer using the environment variable DEFAULT_SETTINGS_REQUESTS_TIMEOUT. Am I missing something?

[dgtlmoon]

@melalj are you talking about webdriver or requests?

[melalj]

@dgtlmoon I'm talking about requests

[melalj]

Here's the error I'm getting:

It looks like the timeout is set at 15 seconds; While, although I know for sure that the endpoint I'm targeting takes more than 20 seconds to resolve.

I have set on the env variables: DEFAULT_SETTINGS_REQUESTS_TIMEOUT=90 (as it was discussed on https://github.com/dgtlmoon/changedetection.io/issues/369)

[melalj]

I'm using the version 0.39.19.1 which supports the timeouts correctly, and all the new versions just fail.

[dgtlmoon]

@melalj not enough information, try to give a concrete example, like which request method, seconds waited, the URL, the exact error, ect

[melalj]

I just did a new test on version 0.40.2 with the URL: https://httpstat.us/200?sleep=85000 while having the env var DEFAULT_SETTINGS_REQUESTS_TIMEOUT=90 - and it seems to be working now.

[freddieleeman]

I'm experiencing the same issue and running v0.41. I have multiple URLs that result in both connect timeout=15 and read timeout=15 errors now.

I've tried adding DEFAULT_SETTINGS_REQUESTS_TIMEOUT=90, but it doesn't change anything.

[freddieleeman]

It appears that hardcoding a timeout here solves the issue: https://github.com/dgtlmoon/changedetection.io/blob/8eee913438c9090e253199867384b7f45756a6d9/changedetectionio/content_fetcher.py#LL729C1-L729C46

So there is an issue somewhere that prevents the variable timeout from being set correctly.

[Turab]

It appears that hardcoding a timeout here solves the issue: https://github.com/dgtlmoon/changedetection.io/blob/8eee913438c9090e253199867384b7f45756a6d9/changedetectionio/content_fetcher.py#LL729C1-L729C46

So there is an issue somewhere that prevents the variable timeout from being set correctly.

This didn't work for me. After I raised this issue as a feature request, @melalj made a pull request and made it possible. Since then I have been using without a problem. Now I needed to move the server. Recreated the container with a new version image, now it breaks at 15 seconds. Setting the env var and even hardcoding doesn't change the result. I go back to versions I know was working; but the outcome is the same.

I don't know if it has anything to do with it but Changedetection doesn't use the system local time in my case. I set the timezone inside the container and then verify it has the correct local time through bash. But Changedetection uses GMT time disregarding the timezone of the system. I don't think these are related but just in case.