Tests hang with Python 3.11

[felixonmars]

Upon updating of Arch Linux's Python from 3.10 to 3.11, I found that mtrpacket's tests hang:

$ ./test.sh
.

I have tried to add some debug outputs and located that the hang is near:

        out_queue.put_nowait('reply ip-6 ::1 round-trip-time 1000')
        result = await mtr.probe('ip6-localhost', ip_version=6)
        command = await in_queue.get()

I have tried to replace ip6-localhost with just localhost and it passes here, but hangs again after await self.send_probes(mtr, in_queue, out_queue).

Any ideas how to proceed here?

[matt-kimball]

I tried Python 3.11 on Ubuntu 22.04, and aside from a deprecation warning, the tests succeed as expected. This leads me to believe this is an Arch specific issue.

I am not an Arch user. What is the easiest way for me to reproduce this issue?

[felixonmars]

Probably inside a docker Arch image?

I have tried to reproduce this hang successfully with Debian sid, but not with Ubuntu focal.

[felixonmars]

BTW, Ubuntu 22.04 comes with Python 3.10 actually. I could reproduce your deprecation warning too with Python 3.10 on Ubuntu 22.04.

I have also tried to install Python 3.11 on Ubuntu 22.04 with ppa:deadsnakes/ppa and run the tests with the install python3.11 command, which reproduces the hang again.

[matt-kimball]

I have installed Python 3.11 from ppa:deadsnakes/ppa, and I cannot reproduce your hang:

mkimball@Alienware:~/mtr-packet-python$ python3.11 --version
Python 3.11.6
mkimball@Alienware:~/mtr-packet-python$ cat test.sh
#!/bin/sh

cd test
PYTHONPATH=.. python3.11 -X dev test.py
mkimball@Alienware:~/mtr-packet-python$ ./test.sh
/home/mkimball/mtr-packet-python/test/test.py:237: DeprecationWarning: There is no current event loop
  loop = asyncio.get_event_loop()
../bin/sh: 1: mtr-packet-missing: not found
..
----------------------------------------------------------------------
Ran 4 tests in 1.096s

OK
mkimball@Alienware:~/mtr-packet-python$

[felixonmars]

It seems to me that exceptions were not handled properly here.

Could you try to add "raise NotImplementedError" right after async def send_probes(self, mtr, in_queue, out_queue):? This makes it hang even earlier here.

[matt-kimball]

You may be right that an exception is occurring somewhere, but so far I am failing to see what you are seeing. Even raising NotImplementedError causes the test to fail and terminate right away, not hang. I will consider what might be going wrong for you though.

[felixonmars]

Not sure if it's related, but Python 3.11.6 fixed a problem that reads similar to me: https://github.com/python/cpython/issues/110894

Unfortunately, I could still reproduce the hang even with the same ppa:deadsnakes/ppa Python 3.11.6 as you.

[lilydjwg]

This is a documented issue:

Note: This method can deadlock when using stdout=PIPE or stderr=PIPE and the child process generates so much output that it blocks waiting for the OS pipe buffer to accept more data. Use the communicate() method when using pipes to avoid this condition.

Simply dropping the await self.process.wait() line will avoid this issue. (But it may cause another issue?)

ip6-localhost doesn't resolve on Arch Linux.

[matt-kimball]

I seriously doubt the issue is the child filling the stdout pipe.

ip6-localhost is much more likely a problem though.

[lilydjwg]

It seems that it's not filling the stdout pipe, but waiting for it to close. The nc subprocess is still alive after .kill()ing the shell process.

[matt-kimball]

Okay - thanks for taking a look.

[lilydjwg]

In this _try_finish method:

https://github.com/python/cpython/blob/31ad7e061ebebc484e00ed3ad5ff61061341c35e/Lib/asyncio/base_subprocess.py#L232-L239

It waits for pipes to close, but the still alive nc process keeps stdout open.