Hundreds of threads after day or more of activity

[undersun81]

Currently I have HAP server running for about 2 weeks and any request just fails because it can't create a new thread.

HAP 2.4.2

gdb attach gdb info thread Shows 254 threads, mostly in recv for example Thread 0x553c7470 (LWP 7985) "python3" 0x76f3e850 in recv () at ../sysdeps/unix/syscall-template.S:84

Here is example of a py-stacktrace <built-in method recv of socket object at remote 0x73dd5978> File "/home/hap/.local/lib/python3.5/site-packages/pyhap/hap_server.py", line 698, in recv block_length_bytes = self.socket.recv(self.LENGTH_LENGTH) File "/home/hap/.local/lib/python3.5/site-packages/pyhap/hap_server.py", line 675, in recv_into data = self.recv(nbytes, flags) File "/usr/lib/python3.5/socket.py", line 576, in readinto return self._sock.recv_into(b) <built-in method readline of _io.BufferedReader object at remote 0x73df2b58> File "/usr/lib/python3.5/http/server.py", line 390, in handle_one_request self.raw_requestline = self.rfile.readline(65537) File "/usr/lib/python3.5/http/server.py", line 424, in handle self.handle_one_request() File "/usr/lib/python3.5/socketserver.py", line 681, in init self.handle() File "/home/hap/.local/lib/python3.5/site-packages/pyhap/hap_server.py", line 143, in init super(HAPServerHandler, self).init(sock, client_addr, server) File "/home/hap/.local/lib/python3.5/site-packages/pyhap/hap_server.py", line 853, in finish_request self, self.accessory_handler) File "/usr/lib/python3.5/socketserver.py", line 625, in process_request_thread self.finish_request(request, client_address) File "/usr/lib/python3.5/threading.py", line 862, in run self._target(*self._args, **self._kwargs) File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner self.run() File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap self._bootstrap_inner()

[undersun81]

I think I have a good guess about what is going on here. Several Apple devices I have at home (iPhones, iPad and HomePod) periodically or by a user request ask HAP-server about state or subscribe. From logs I can see they use different client ports, but they(clients) never close these sockets and next time use another port and HAP-server creates new connection and new corresponding thread. Linux by default has very long timeout values for TCP-sockets, so eventually I have hundreds of threads and open connections. So I hit the limit somewhere in python and it just fails with error when trying to create new thread. This story happen to me every evening, then during night most of sockets timed out and I had proper usage during the day, and story repeated in the evening.

HAP uses blocking sockets, so my solution was to set timeout for a client socket when we connected. So in hap_server.py in method HAPServer.get_request I added "client_socket.settimeout(60)" after "client_socket, client_addr = super(HAPServer, self).get_request()".

This change has fixed my problem, but I'm not an expert in HomeKit protocol and not sure about subscription case. May be clients which subscribed expect to hold connection as long as possible. If so then they should send some ping periodically into the same connection and not just start talking through another socket. Looks like apple devices just don't care to properly shutdown those sockets (may be in some specific case, like sleeping device) and they seem don't care if we have short timeout on them.

[niobos]

I don't think it has to do with open connections. I have TCP-keepalive configured, so dead connections get killed after about 5 minutes. Still, I see the number of threads increasing over time...