hydrusnetwork
[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1007)
Hydrus version
550
Qt major version
Qt 6
Operating system
Linux (specify distro and version in comments)
Install method
Installer
Install and OS comments
Ubuntu 22.04.3 LTS
Bug description and reproduction
One or two files fail to download from certain providers even when those files load just fine in every browser. From my cursory googling, it is an SSL error that stems from either (1) the server's choice of params or (2) the client Python urllib3 library. Since the same source URLs load just fine in every browser, it seems like a client/Hydrus issue. Either way, we should have an option to side-step/disable this cipher check and/or ignore any SSL-related error when manually reviewing failed downloads.
Thanks!
Log output
ConnectionException('Problem with SSL: SSLError(MaxRetryError("HTTPSConnectionPool(host=\'XYZW\', port=X): Max retries exceeded with url: /XYZW (Caused by SSLError(SSLError(1, \'[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1007)\')))"))')… (Copy note to see full error)
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 468, in _make_request
File "urllib3/connectionpool.py", line 1097, in _validate_conn
File "urllib3/connection.py", line 642, in connect
File "urllib3/connection.py", line 783, in _ssl_wrap_socket_and_match_hostname
File "urllib3/util/ssl_.py", line 471, in ssl_wrap_socket
File "urllib3/util/ssl_.py", line 515, in _ssl_wrap_socket_impl
File "ssl.py", line 513, in wrap_socket
File "ssl.py", line 1104, in _create
File "ssl.py", line 1375, in do_handshake
ssl.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1007)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "urllib3/connectionpool.py", line 791, in urlopen
File "urllib3/connectionpool.py", line 492, in _make_request
urllib3.exceptions.SSLError: [SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1007)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "requests/adapters.py", line 486, in send
File "urllib3/connectionpool.py", line 845, in urlopen
File "urllib3/util/retry.py", line 515, in increment
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='XYZW', port=X): Max retries exceeded with url: /XYZW (Caused by SSLError(SSLError(1, '[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1007)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "hydrus/client/networking/ClientNetworkingJobs.py", line 1555, in Start
File "hydrus/client/networking/ClientNetworkingJobs.py", line 792, in _SendRequestAndGetResponse
File "requests/sessions.py", line 589, in request
File "requests/sessions.py", line 703, in send
File "requests/adapters.py", line 517, in send
requests.exceptions.SSLError: HTTPSConnectionPool(host='XYZW', port=X): Max retries exceeded with url: /XYZW (Caused by SSLError(SSLError(1, '[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1007)')))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "hydrus/client/importing/ClientImportFileSeeds.py", line 1528, in WorkOnURL
File "hydrus/client/importing/ClientImportFileSeeds.py", line 662, in DownloadAndImportRawFile
File "hydrus/client/networking/ClientNetworkingJobs.py", line 1977, in WaitUntilDone
File "hydrus/client/networking/ClientNetworkingJobs.py", line 1737, in Start
hydrus.core.HydrusExceptions.ConnectionException: Problem with SSL: SSLError(MaxRetryError("HTTPSConnectionPool(host='XYZW', port=X): Max retries exceeded with url: /XYZW (Caused by SSLError(SSLError(1, '[SSL: DH_KEY_TOO_SMALL] dh key too small (_ssl.c:1007)')))"))
Thanks for this. I have seen a couple of similar reports. I agree, I think this is probably because we are still on Python 3.10 for the official builds, and perhaps combined with our aging network engine, requests, which is still on HTTP 1.1 iirc, although I don't know if that interferes with the SSL negotiation here, so maybe not. Unfortunately, I do not believe this is a bug I can simply set some sort of verify=false to, since it is in the handshake stage and it is, I think, the server asking us to do something our version of SSL isn't happy with, or it is the server sending this error to our end. At least, afaik. There is a BUGFIX setting under options->connection that allows you to not verify regular https--feel free to try it and let me know how you get on.
I would like to move up to 3.11 fairly soon, although I absolutely need to put out a 'future build' release first so people can test. I tried to do this a few weeks ago but simply updating the library versions ran afoul of some PyInstaller issues, on Windows with how modern PyInstaller includes certain files and on Linux it simply failed, either due to PyInstaller or PySide6 version. I think I know what I need to do, it'll just take work, and in the outside case, a move to pyoxidizer.
I don't know when I will reattempt this work, but it is due in my immediate to-do list, so I hope fairly soon. I'll post the links in my normal release post, so you can try it yourself as well.
If you need a faster fix, and you have access to python 3.11 on your system, I recommend you try running from source. It is much easier to set up these days, just a couple of lines in terminal and you are running: https://hydrusnetwork.github.io/hydrus/running_from_source.html
Thanks for the detailed response! I may try running from source (I have a couple ideas I want to try my hand at anyway), though I did find a workaround by forcing the provider to serve from a different network. I'll keep that BUGFIX option in mind going forward :)
And by the way, this is truly a wonderful piece of software. Thank you for your work on this.