ssl.SSLZeroReturnError exception on startup with builtin ssl backend python 3.8 or above

[toppk]

❓ I'm submitting a ...

• [X] 🐞 bug report
• [ ] 🐣 feature request
• [ ] ❓ question about the decisions made in the repository

🐞 Describe the bug. What is the current behavior? I get an exception on startup. Everything seems to be working okay. I have attached the exception, and the code used you will need to generate an ssl cert/key pair to test.

❓ What is the motivation / use case for changing the behavior?

💡 To Reproduce

python test.py

this is the exception so people don't have to download the attachment. the code is a basic CherryPy startup that I grabbed from https://github.com/cherrypy/cheroot/issues/346

[17/Sep/2022:01:17:23] ENGINE Started monitor thread 'Autoreloader'.
[17/Sep/2022:01:17:23] ENGINE Serving on https://0.0.0.0:4400
[17/Sep/2022:01:17:23] ENGINE Bus STARTED
[17/Sep/2022:01:17:23] ENGINE Error in HTTPServer.serve
Traceback (most recent call last):
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve
    self._connections.run(self.expiration_interval)
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run
    self._run(expiration_interval)
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run
    new_conn = self._from_server_socket(self.server.socket)
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket
    s, ssl_env = self.server.ssl_adapter.wrap(s)
  File "/home/toppk/.local/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap
    s = self.context.wrap_socket(
  File "/usr/lib64/python3.9/ssl.py", line 501, in wrap_socket
    return self.sslsocket_class._create(
  File "/usr/lib64/python3.9/ssl.py", line 1041, in _create
    self.do_handshake()
  File "/usr/lib64/python3.9/ssl.py", line 1310, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129)

exception.txt test.py.txt

💡 Expected behavior startup without exception

📋 Details

📋 Environment

• Cheroot version: 8.6.0
• CherryPy version: 18.6.0
• Python version: 3.9.14
• OS: Fedora 37 beta
• Browser: all

📋 Additional context I've tried this in python3.11rc2 and python3.9, both have the issue. I do not see any similar issue if I switch to pyopenssl backend.

[toppk]

one thing is important to note, is that there is no tcp connection made when the exception occurs, this exception is spontaneous.

[webknjaz]

This strongly reminds me of https://github.com/cherrypy/cherrypy/issues/1618.

[webknjaz]

one thing is important to note, is that there is no tcp connection made when the exception occurs, this exception is spontaneous.

Well, that might be CherryPy's Checker kicking in. Have you tried disabling it?

[toppk]

i disabled the checker, no difference. I tested on ubuntu 22.04 with python3.10 and the same issue.

[webknjaz]

Oh, wait, I remember there was some kind of a TLS probe added in relatively recent releases. Maybe that's what's failing...

FWIW It'd be useful to have a pure-Cheroot repro for this.

[toppk]

update: tried straight cheroot and it didn't have any issues, added logging and I can definetly see the peer address so it may be a tls probe (which I haven't found in the code just yet).

Another major update, is that this issue goes away when using python<3.8, no changes in cherrypy/cheroot versions, just switching between python 3.8 and python 3.7 flips this exception on and off, and they both seem to be hitting the same codepath (connection.run())

[toppk]

Okay, I monkey patched socket, and printed a stacktrace on connect and found the code. it is portend, specifically in cherrypy/process/servers.py

# wait for port to be occupied
with _safe_wait(*self.bound_addr):
        portend.occupied(*self.bound_addr, timeout=Timeouts.occupied)

So everything is happening as expected. There are two questions.

1. why does cheroot under python 3.7 hid these zero-bytes-sent connections and only see them under python 3.8, and
2. do we want to hide such exceptions in similar cases in the future?

I can confirm that just telnet'ing to the port and closing the connection will generate this exception under python>=3.8 and not log anything otherwise.

I personally can understand leaving this issue alone, but I think it's bad as is, especially since we forcibly generate this corner case on startup. I will try to understand how python 3.7 is able to silence these 0byte tcp connections.

[toppk]

adding the test code w/ monkeypatch, in case people need to debug something like this in the future.

import cherrypy
import sys

import socket

print("python version:", sys.version)
print("cherrpy version:", cherrypy.__version__)


class RootServer:
    @cherrypy.expose
    def index(self, **keywords):
        return "it works!"


oldSocket = socket.socket


class MonkeySocket2(oldSocket):
    def __init__(self, *args, **kwargs):
        super().__init__(*args, **kwargs)

    def connect(self, *args, **kwargs):
        import traceback
        traceback.print_stack()
        print("in connect")
        return super().connect(*args, **kwargs)

socket.socket = MonkeySocket2

if __name__ == "__main__":
    server_config = {
        #'server.socket_host': '127.0.0.1',
        "server.socket_host": "0.0.0.0",
        "server.socket_port": 4400,
        "server.ssl_module": "builtin",
        #'server.ssl_module':'pyopenssl',
        "server.ssl_certificate": "local/cert.pem",
        "server.ssl_private_key": "local/privkey.pem",
        "engine.autoreload.on": False,
        "checker.on": False,
    }

    cherrypy.config.update(server_config)
    cherrypy.quickstart(RootServer())

[toppk]

I found the change in cpython that caused the issue, and submitted a pr to cheroot

[toppk]

This strongly reminds me of cherrypy/cherrypy#1618.

as it turns out, the cpython patch was made in response to cherrypy/cherrypy#1618 , which ended up manifesting the same exception a different way. This was a confusing bug to track down.

[The-Compiler]

I started seeing this as well after a recent Archlinux system update. I'm not entirely sure that the CPython commit linked in #518 is really to blame (after all, it also talks about ssl.SSLEOFError and not SSLZeroReturnError!).

The more likely culprit seems to be upgraded openssl (1.1.1.q-1 -> 3.0.7-2) and not any change in CPython.

[thezoggy]

seeing more people report seeing this more often now after their os upgraded to using openssl 3.x,

 ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:997)

[systemdarena]

is there any workaround to just hide these SSLZeroReturnError exceptions in the mean time?

Note I am running this through a pretty standard cherrypy setup with a cherrypy.engine.start() and cherrypy.engine.block()

thanks

[vishvasAshrivats]

I am facing the issue when starting/restarting the cherrypy server, i have checked through these links as well, and understand the merge is blocked on 518

• https://forums.sabnzbd.org/viewtopic.php?t=26216
• https://github.com/cherrypy/cherrypy/issues/1618
• https://github.com/cherrypy/cheroot/pull/518
• https://github.com/python/cpython/pull/18772

Here is the error i am facing ,

Error in HTTPServer.tick
Traceback (most recent call last):
  File "cheroot\server.py", line 1770, in serve
  File "cheroot\server.py", line 1993, in tick
  File "cheroot\connections.py", line 180, in get_conn
  File "cheroot\connections.py", line 199, in _from_server_socket
  File "cheroot\ssl\builtin.py", line 113, in wrap
  File "ssl.py", line 517, in wrap_socket
  File "ssl.py", line 1075, in _create
  File "ssl.py", line 1346, in do_handshake
ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:992)

Version information:

• Python Version: 3.11.2
• cheroot==8.2.1
• cherrypy==18.8.0

is there any information on the ETA, or possible fix on this?

Thanks in Advance