Workaround uvloop losing track of sockets when passing a socket to create_connection
What do these changes do?
Workaround uvloop losing track of sockets when passing a socket to create_connection.
As we are coming up on a year of https://github.com/MagicStack/uvloop/issues/645, and the attempt to fix it https://github.com/MagicStack/uvloop/pull/646 being open, it appears the issue is not going to be fixed in uvloop soon.
related issue #10506 where this solution has been tested
Are there changes in behavior for the user?
Is it a substantial burden for the maintainers to support this?
Related issue number
Checklist
- [ ] I think the code is well written
- [ ] Unit tests for the changes exist
- [ ] Documentation reflects the changes
- [ ] If you provide code modification, please add yourself to
CONTRIBUTORS.txt- The format is <Name> <Surname>.
- Please keep alphabetical order, the file is sorted by names.
- [ ] Add a new news fragment into the
CHANGES/folder-
name it
<issue_or_pr_num>.<type>.rst(e.g.588.bugfix.rst) -
if you don't have an issue number, change it to the pull request number after creating the PR
.bugfix: A bug fix for something the maintainers deemed an improper undesired behavior that got corrected to match pre-agreed expectations..feature: A new behavior, public APIs. That sort of stuff..deprecation: A declaration of future API removals and breaking changes in behavior..breaking: When something public is removed in a breaking way. Could be deprecated in an earlier release..doc: Notable updates to the documentation structure or build process..packaging: Notes for downstreams about unobvious side effects and tooling. Changes in the test invocation considerations and runtime assumptions..contrib: Stuff that affects the contributor experience. e.g. Running tests, building the docs, setting up the development environment..misc: Changes that are hard to assign to any of the above categories.
-
Make sure to use full sentences with correct case and punctuation, for example:
Fixed issue with non-ascii contents in doctest text files -- by :user:`contributor-gh-handle`.Use the past tense or the present tense a non-imperative mood, referring to what's changed compared to the last released version of this project.
-
This still needs tests
:x: 1 Tests Failed:
| Tests completed | Failed | Passed | Skipped |
|---|---|---|---|
| 4228 | 1 | 4227 | 44 |
View the full list of 1 :snowflake: flaky test(s)
tests.test_connector::test_tcp_connector_happy_eyeballs[pyloop-None]Flake rate in main: 6.67% (Passed 28 times, Failed 2 times)
Stack Traces | 0.103s run time
loop = <_UnixSelectorEventLoop running=False closed=False debug=False> happy_eyeballs_delay = None #x1B[0m#x1B[37m@pytest#x1B[39;49;00m.mark.parametrize(#x1B[90m#x1B[39;49;00m (#x1B[33m"#x1B[39;49;00m#x1B[33mhappy_eyeballs_delay#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m [#x1B[94m0.1#x1B[39;49;00m, #x1B[94m0.25#x1B[39;49;00m, #x1B[94mNone#x1B[39;49;00m],#x1B[90m#x1B[39;49;00m )#x1B[90m#x1B[39;49;00m #x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mtest_tcp_connector_happy_eyeballs#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m loop: asyncio.AbstractEventLoop, happy_eyeballs_delay: Optional[#x1B[96mfloat#x1B[39;49;00m]#x1B[90m#x1B[39;49;00m ) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m conn = aiohttp.TCPConnector(happy_eyeballs_delay=happy_eyeballs_delay)#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m ip1 = #x1B[33m"#x1B[39;49;00m#x1B[33mdead::beef::#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m ip2 = #x1B[33m"#x1B[39;49;00m#x1B[33m192.168.1.1#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m#x1B[90m#x1B[39;49;00m ips = [ip1, ip2]#x1B[90m#x1B[39;49;00m addrs_tried = []#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m req = ClientRequest(#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33mGET#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m URL(#x1B[33m"#x1B[39;49;00m#x1B[33mhttps://mocked.host#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m),#x1B[90m#x1B[39;49;00m loop=loop,#x1B[90m#x1B[39;49;00m )#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m #x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92m_resolve_host#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m host: #x1B[96mstr#x1B[39;49;00m, port: #x1B[96mint#x1B[39;49;00m, traces: #x1B[96mobject#x1B[39;49;00m = #x1B[94mNone#x1B[39;49;00m#x1B[90m#x1B[39;49;00m ) -> List[ResolveResult]:#x1B[90m#x1B[39;49;00m #x1B[94mreturn#x1B[39;49;00m [#x1B[90m#x1B[39;49;00m {#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33mhostname#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: host,#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33mhost#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: ip,#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33mport#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: port,#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33mfamily#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: socket.AF_INET6 #x1B[94mif#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33m:#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m #x1B[95min#x1B[39;49;00m ip #x1B[94melse#x1B[39;49;00m socket.AF_INET,#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33mproto#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: #x1B[94m0#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33mflags#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m: socket.AI_NUMERICHOST,#x1B[90m#x1B[39;49;00m }#x1B[90m#x1B[39;49;00m #x1B[94mfor#x1B[39;49;00m ip #x1B[95min#x1B[39;49;00m ips#x1B[90m#x1B[39;49;00m ]#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m os_error = #x1B[94mFalse#x1B[39;49;00m#x1B[90m#x1B[39;49;00m connected = #x1B[94mFalse#x1B[39;49;00m#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m #x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92msock_connect#x1B[39;49;00m(*args: Tuple[#x1B[96mstr#x1B[39;49;00m, #x1B[96mint#x1B[39;49;00m], **kwargs: #x1B[96mobject#x1B[39;49;00m) -> #x1B[94mNone#x1B[39;49;00m:#x1B[90m#x1B[39;49;00m addr = args[#x1B[94m1#x1B[39;49;00m]#x1B[90m#x1B[39;49;00m #x1B[94mnonlocal#x1B[39;49;00m os_error#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m addrs_tried.append(addr)#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m #x1B[94mif#x1B[39;49;00m addr[#x1B[94m0#x1B[39;49;00m] == ip1:#x1B[90m#x1B[39;49;00m os_error = #x1B[94mTrue#x1B[39;49;00m#x1B[90m#x1B[39;49;00m #x1B[94mraise#x1B[39;49;00m #x1B[96mOSError#x1B[39;49;00m#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m #x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mcreate_connection#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m *args: #x1B[96mobject#x1B[39;49;00m, sock: Optional[socket.socket] = #x1B[94mNone#x1B[39;49;00m, **kwargs: #x1B[96mobject#x1B[39;49;00m#x1B[90m#x1B[39;49;00m ) -> Tuple[ResponseHandler, ResponseHandler]:#x1B[90m#x1B[39;49;00m #x1B[94massert#x1B[39;49;00m #x1B[96misinstance#x1B[39;49;00m(sock, socket.socket)#x1B[90m#x1B[39;49;00m #x1B[90m# Close the socket since we are not actually connecting#x1B[39;49;00m#x1B[90m#x1B[39;49;00m #x1B[90m# and we don't want to leak it.#x1B[39;49;00m#x1B[90m#x1B[39;49;00m sock.close()#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m #x1B[94mnonlocal#x1B[39;49;00m connected#x1B[90m#x1B[39;49;00m connected = #x1B[94mTrue#x1B[39;49;00m#x1B[90m#x1B[39;49;00m tr = create_mocked_conn(loop)#x1B[90m#x1B[39;49;00m pr = create_mocked_conn(loop)#x1B[90m#x1B[39;49;00m #x1B[94mreturn#x1B[39;49;00m tr, pr#x1B[90m#x1B[39;49;00m #x1B[90m#x1B[39;49;00m #x1B[94mwith#x1B[39;49;00m mock.patch.object(#x1B[90m#x1B[39;49;00m conn, #x1B[33m"#x1B[39;49;00m#x1B[33m_resolve_host#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m, autospec=#x1B[94mTrue#x1B[39;49;00m, spec_set=#x1B[94mTrue#x1B[39;49;00m, side_effect=_resolve_host#x1B[90m#x1B[39;49;00m ):#x1B[90m#x1B[39;49;00m #x1B[94mwith#x1B[39;49;00m mock.patch.object(#x1B[90m#x1B[39;49;00m conn._loop,#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33msock_connect#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m autospec=#x1B[94mTrue#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m spec_set=#x1B[94mTrue#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m side_effect=sock_connect,#x1B[90m#x1B[39;49;00m ):#x1B[90m#x1B[39;49;00m #x1B[94mwith#x1B[39;49;00m mock.patch.object(#x1B[90m#x1B[39;49;00m conn._loop,#x1B[90m#x1B[39;49;00m #x1B[33m"#x1B[39;49;00m#x1B[33mcreate_connection#x1B[39;49;00m#x1B[33m"#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m autospec=#x1B[94mTrue#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m spec_set=#x1B[94mTrue#x1B[39;49;00m,#x1B[90m#x1B[39;49;00m side_effect=create_connection,#x1B[90m#x1B[39;49;00m ):#x1B[90m#x1B[39;49;00m > established_connection = #x1B[94mawait#x1B[39;49;00m conn.connect(req, [], ClientTimeout())#x1B[90m#x1B[39;49;00m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m _resolve_host = <function test_tcp_connector_happy_eyeballs.<locals>._resolve_host at 0x7fb52d02a020> addrs_tried = [] conn = <aiohttp.connector.TCPConnector object at 0x7fb52bc6f310> connected = False create_connection = <function test_tcp_connector_happy_eyeballs.<locals>.create_connection at 0x7fb52d029da0> happy_eyeballs_delay = None ip1 = 'dead::beef::' ip2 = '192.168.1.1' ips = ['dead::beef::', '192.168.1.1'] loop = <_UnixSelectorEventLoop running=False closed=False debug=False> os_error = False req = <aiohttp.client_reqrep.ClientRequest object at 0x7fb52bc6f510> sock_connect = <function test_tcp_connector_happy_eyeballs.<locals>.sock_connect at 0x7fb52d02b100> #x1B[1m#x1B[31mtests/test_connector.py#x1B[0m:966: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ #x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:599: in connect #x1B[0mproto = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._create_connection(req, traces, timeout)#x1B[90m#x1B[39;49;00m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m conn = None key = ConnectionKey(host='mocked.host', port=443, is_ssl=True, ssl=True, proxy=None, proxy_auth=None, proxy_headers_hash=None) placeholder = <aiohttp.connector._TransportPlaceholder object at 0x7fb52ceb1b40> req = <aiohttp.client_reqrep.ClientRequest object at 0x7fb52bc6f510> self = <aiohttp.connector.TCPConnector object at 0x7fb52bc6f310> timeout = ClientTimeout(total=None, connect=None, sock_read=None, sock_connect=None, ceil_threshold=5) traces = [] #x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:1163: in _create_connection #x1B[0m_, proto = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._create_direct_connection(req, traces, timeout)#x1B[90m#x1B[39;49;00m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m req = <aiohttp.client_reqrep.ClientRequest object at 0x7fb52bc6f510> self = <aiohttp.connector.TCPConnector object at 0x7fb52bc6f310> timeout = ClientTimeout(total=None, connect=None, sock_read=None, sock_connect=None, ceil_threshold=5) traces = [] #x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:1457: in _create_direct_connection #x1B[0mtransp, proto = #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._wrap_create_connection(#x1B[90m#x1B[39;49;00m addr_infos = [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('dead::beef::', 443, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('192.168.1.1', 443))] client_error = <class 'aiohttp.client_exceptions.ClientConnectorError'> fingerprint = None host = 'mocked.host' hosts = [{'family': <AddressFamily.AF_INET6: 10>, 'flags': <AddressInfo.AI_NUMERICHOST: 4>, 'host': 'dead::beef::', 'hostname'...ssFamily.AF_INET: 2>, 'flags': <AddressInfo.AI_NUMERICHOST: 4>, 'host': '192.168.1.1', 'hostname': 'mocked.host', ...}] last_exc = None port = 443 req = <aiohttp.client_reqrep.ClientRequest object at 0x7fb52bc6f510> self = <aiohttp.connector.TCPConnector object at 0x7fb52bc6f310> server_hostname = 'mocked.host' sslcontext = <ssl.SSLContext object at 0x7fb53416f140> timeout = ClientTimeout(total=None, connect=None, sock_read=None, sock_connect=None, ceil_threshold=5) traces = [] #x1B[1m#x1B[31maiohttp/connector.py#x1B[0m:1241: in _wrap_create_connection #x1B[0m#x1B[94mreturn#x1B[39;49;00m #x1B[94mawait#x1B[39;49;00m #x1B[96mself#x1B[39;49;00m._loop.create_connection(#x1B[90m#x1B[39;49;00m addr_infos = [(<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('dead::beef::', 443, 0, 0)), (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('192.168.1.1', 443))] address_tuple = ('dead::beef::', 443, 0, 0) args = (functools.partial(<class 'aiohttp.client_proto.ResponseHandler'>, loop=<_UnixSelectorEventLoop running=False closed=False debug=False>),) client_error = <class 'aiohttp.client_exceptions.ClientConnectorError'> first_addr_infos = (<AddressFamily.AF_INET6: 10>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('dead::beef::', 443, 0, 0)) host = 'dead::beef::' kwargs = {'server_hostname': 'mocked.host', 'ssl': <ssl.SSLContext object at 0x7fb53416f140>} port = 443 req = <aiohttp.client_reqrep.ClientRequest object at 0x7fb52bc6f510> self = <aiohttp.connector.TCPConnector object at 0x7fb52bc6f310> timeout = ClientTimeout(total=None, connect=None, sock_read=None, sock_connect=None, ceil_threshold=5) #x1B[1m#x1B[.../hostedtoolcache/Python/3.11.13.../x64/lib/python3.11/unittest/mock.py#x1B[0m:2251: in _execute_mock_call #x1B[0mresult = #x1B[94mawait#x1B[39;49;00m effect(*args, **kwargs)#x1B[90m#x1B[39;49;00m ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^#x1B[90m#x1B[39;49;00m _call = call(functools.partial(<class 'aiohttp.client_proto.ResponseHandler'>, loop=<_UnixSelectorEventLoop running=False clos...g=False>), host='dead::beef::', port=443, ssl=<ssl.SSLContext object at 0x7fb53416f140>, server_hostname='mocked.host') args = (functools.partial(<class 'aiohttp.client_proto.ResponseHandler'>, loop=<_UnixSelectorEventLoop running=False closed=False debug=False>),) effect = <function test_tcp_connector_happy_eyeballs.<locals>.create_connection at 0x7fb52d029da0> kwargs = {'host': 'dead::beef::', 'port': 443, 'server_hostname': 'mocked.host', 'ssl': <ssl.SSLContext object at 0x7fb53416f140>} self = <AsyncMock name='create_connection' spec_set='method' id='140416118651152'> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ sock = None args = (functools.partial(<class 'aiohttp.client_proto.ResponseHandler'>, loop=<_UnixSelectorEventLoop running=False closed=False debug=False>),) kwargs = {'host': 'dead::beef::', 'port': 443, 'server_hostname': 'mocked.host', 'ssl': <ssl.SSLContext object at 0x7fb53416f140>} @py_assert3 = <class 'socket.socket'>, @py_assert5 = False @py_format7 = "assert False\n{False = isinstance(None, <class 'socket.socket'>\n{<class 'socket.socket'> = socket.socket\n})\n}" #x1B[0m#x1B[94masync#x1B[39;49;00m #x1B[94mdef#x1B[39;49;00m#x1B[90m #x1B[39;49;00m#x1B[92mcreate_connection#x1B[39;49;00m(#x1B[90m#x1B[39;49;00m *args: #x1B[96mobject#x1B[39;49;00m, sock: Optional[socket.socket] = #x1B[94mNone#x1B[39;49;00m, **kwargs: #x1B[96mobject#x1B[39;49;00m#x1B[90m#x1B[39;49;00m ) -> Tuple[ResponseHandler, ResponseHandler]:#x1B[90m#x1B[39;49;00m > #x1B[94massert#x1B[39;49;00m #x1B[96misinstance#x1B[39;49;00m(sock, socket.socket)#x1B[90m#x1B[39;49;00m #x1B[1m#x1B[31mE AssertionError: assert False#x1B[0m #x1B[1m#x1B[31mE + where False = isinstance(None, <class 'socket.socket'>)#x1B[0m #x1B[1m#x1B[31mE + where <class 'socket.socket'> = socket.socket#x1B[0m args = (functools.partial(<class 'aiohttp.client_proto.ResponseHandler'>, loop=<_UnixSelectorEventLoop running=False closed=False debug=False>),) connected = False kwargs = {'host': 'dead::beef::', 'port': 443, 'server_hostname': 'mocked.host', 'ssl': <ssl.SSLContext object at 0x7fb53416f140>} loop = <_UnixSelectorEventLoop running=False closed=False debug=False> sock = None #x1B[1m#x1B[31mtests/test_connector.py#x1B[0m:938: AssertionError
To view more test analytics, go to the Test Analytics Dashboard 📋 Got 3 mins? Take this short survey to help us improve Test Analytics.
CodSpeed Performance Report
Merging #11539 will not alter performance
Comparing single_create_connection_call_happy_eyeballs_disabled (decb65c) with master (2518c59)
Summary
✅ 59 untouched
This causes widespread localhost service access failures, aiohappyeyeballs defaults to IPv6-first resolution, but most localhost services only listen on IPv4.
This causes widespread localhost service access failures, aiohappyeyeballs defaults to IPv6-first resolution, but most localhost services only listen on IPv4.
Are you saying it causes failures when happyeyeballs is enabled? I would expect the behavior to be the same when its disabled
This causes widespread localhost service access failures, aiohappyeyeballs defaults to IPv6-first resolution, but most localhost services only listen on IPv4.
Are you saying it causes failures when happyeyeballs is enabled? I would expect the behavior to be the same when its disabled
only cause when happyeyeballs is disabled. weather happyeyeballs is enabled or disabled. resolved addrinfo always comes from happyeyeballs and IPV6 first.
first_addr_infos = addr_infos[0]
address_tuple = first_addr_infos[4]
host: str = address_tuple[0]
port: int = address_tuple[1]
when IPv6 is enabled, localhost will resolved to ::1 first and http service is not listening at IPv6. and then connection will be refused.
resolved addrinfo always comes from happyeyeballs and IPV6 first.
I'm lost on this comment as happyeyeballs doesn't do the resolution, the configured resolver does
does
my mistake,it's os related. but always only get first address when happyeyeballs disabled will broken localhost ipv4 service under some os as ubuntu.
It will eventually fallback to the next address if the timeout is long enough, which is the pre-happyeyeballs behavior. In Home Assistant we used to set the family to AF_INET before we had happyeyeballs to disable IPv6
is winloop is also affected by this or is it just uvloop?
Any update? Maybe I prefer to do a monkey patch, if it will cost much more time to merge this fix.