ResourceWarning: unclosed event loop on macOS, within pytest

[rdbisme]

trafficstars

I just updated command line tools and macOS to 10.15.4, I'm within a virtualenv, using ipdb within pytest, Python v. 3.7.7. Don't know if the update is related.

I'm getting the following error:

Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 622, in __del__
    source=self)
ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
Exception ignored in: <socket.socket fd=8, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
Exception ignored in: <socket.socket fd=9, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
ResourceWarning: unclosed <socket.socket fd=9, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
Exception ignored in: <function BaseEventLoop.__del__ at 0x11b5d73b0>
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 622, in __del__
    source=self)
ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
Exception ignored in: <socket.socket fd=32, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
ResourceWarning: unclosed <socket.socket fd=32, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
Exception ignored in: <socket.socket fd=33, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
ResourceWarning: unclosed <socket.socket fd=33, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
Exception ignored in: <function BaseEventLoop.__del__ at 0x11b5d73b0>
Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/asyncio/base_events.py", line 622, in __del__
    source=self)
ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
Exception ignored in: <socket.socket fd=46, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
ResourceWarning: unclosed <socket.socket fd=46, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
Exception ignored in: <socket.socket fd=47, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
ResourceWarning: unclosed <socket.socket fd=47, family=AddressFamily.AF_UNIX, type=SocketKind.SOCK_STREAM, proto=0>
Exception ignored in: <function BaseEventLoop.__del__ at 0x11b5d73b0>
Traceback (most recent call last):

Randomly in the ipdb prompt.

[Carreau]

Damn. Thanks.

If you can reproduce easily and dont' mind enabling tracemalloc that should change the error message to tell you where this eventloop was created; which should help to debug.

[rdbisme]

It looks like the problem vanished again. :/.

If it comes out again I'll try to give it a deeper look. Thanks!

[mattiasa]

I'm hitting this too and have reproduced it in a small test project available at https://github.com/mattiasa/ipython-issue-12203/

I'm reproducing this on OSX 10.14.6 with both Python 3.6.8 and 3.8.3.

To reproduce:

• Create a virtual env
• Install requirements.txt (I froze my entire environment)
• Run python test.py
• Step through a few lines of code:

: lapetus ipython (master) \$ ; virtualenv -p python3.8 env
Running virtualenv with interpreter /usr/local/bin/python3.8
Already using interpreter /usr/local/bin/python3.8
Using base prefix '/Library/Frameworks/Python.framework/Versions/3.8'
New python executable in /Users/mattiasa/test/ipython/env/bin/python3.8
Also creating executable in /Users/mattiasa/test/ipython/env/bin/python
Installing setuptools, pip, wheel...
done.
: lapetus ipython (master) \$ ; . env/bin/activate
(env) : lapetus ipython (master) \$ ; pip install -r requirements.txt
Collecting appnope==0.1.0
  Using cached appnope-0.1.0-py2.py3-none-any.whl (4.0 kB)
Collecting backcall==0.1.0
  Using cached backcall-0.1.0.tar.gz (9.7 kB)
Collecting decorator==4.4.2
  Using cached decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
Collecting ipdb==0.13.2
  Using cached ipdb-0.13.2.tar.gz (14 kB)
Collecting ipython==7.14.0
  Using cached ipython-7.14.0-py3-none-any.whl (782 kB)
Collecting ipython-genutils==0.2.0
  Using cached ipython_genutils-0.2.0-py2.py3-none-any.whl (26 kB)
Collecting jedi==0.17.0
  Using cached jedi-0.17.0-py2.py3-none-any.whl (1.1 MB)
Collecting parso==0.7.0
  Using cached parso-0.7.0-py2.py3-none-any.whl (100 kB)
Collecting pexpect==4.8.0
  Using cached pexpect-4.8.0-py2.py3-none-any.whl (59 kB)
Collecting pickleshare==0.7.5
  Using cached pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)
Collecting prompt-toolkit==3.0.5
  Using cached prompt_toolkit-3.0.5-py3-none-any.whl (351 kB)
Collecting ptyprocess==0.6.0
  Using cached ptyprocess-0.6.0-py2.py3-none-any.whl (39 kB)
Collecting Pygments==2.6.1
  Using cached Pygments-2.6.1-py3-none-any.whl (914 kB)
Collecting six==1.15.0
  Using cached six-1.15.0-py2.py3-none-any.whl (10 kB)
Collecting traitlets==4.3.3
  Using cached traitlets-4.3.3-py2.py3-none-any.whl (75 kB)
Collecting wcwidth==0.1.9
  Using cached wcwidth-0.1.9-py2.py3-none-any.whl (19 kB)
Requirement already satisfied: setuptools in ./env/lib/python3.8/site-packages (from ipdb==0.13.2->-r requirements.txt (line 4)) (46.4.0)
Building wheels for collected packages: backcall, ipdb
  Building wheel for backcall (setup.py) ... done
  Created wheel for backcall: filename=backcall-0.1.0-py3-none-any.whl size=10413 sha256=b47d4904f77ce923e5c68ef8f1f4e847ef789e48f34a60ec5c1f53de7ee6781e
  Stored in directory: /Users/mattiasa/Library/Caches/pip/wheels/19/62/a8/be9667e6ae351443886ae17237f574a2c265d1ad27ec22c7dc
  Building wheel for ipdb (setup.py) ... done
  Created wheel for ipdb: filename=ipdb-0.13.2-py3-none-any.whl size=10520 sha256=39cda699365d01a09d7266672dc434483487b94805b55f6b7568f1fcec54c6de
  Stored in directory: /Users/mattiasa/Library/Caches/pip/wheels/31/f3/58/accdf45cba573136dff3bc0e8aad1848899255f5fc3266a9fc
Successfully built backcall ipdb
Installing collected packages: appnope, backcall, decorator, pickleshare, parso, jedi, six, ipython-genutils, traitlets, wcwidth, prompt-toolkit, ptyprocess, pexpect, Pygments, ipython, ipdb
Successfully installed Pygments-2.6.1 appnope-0.1.0 backcall-0.1.0 decorator-4.4.2 ipdb-0.13.2 ipython-7.14.0 ipython-genutils-0.2.0 jedi-0.17.0 parso-0.7.0 pexpect-4.8.0 pickleshare-0.7.5 prompt-toolkit-3.0.5 ptyprocess-0.6.0 six-1.15.0 traitlets-4.3.3 wcwidth-0.1.9
(env) : lapetus ipython (master) \$ ; python3.8 test.py
DEBUG:asyncio:Using selector: KqueueSelector
DEBUG:asyncio:Using selector: KqueueSelector
> /Users/mattiasa/test/ipython/test.py(6)test()
      5         import ipdb; ipdb.set_trace()
----> 6         for i in range(1, 10):
      7             self.assertEqual(i, i)

DEBUG:asyncio:Using selector: KqueueSelector
ipdb> n
> /Users/mattiasa/test/ipython/test.py(7)test()
      6         for i in range(1, 10):
----> 7             self.assertEqual(i, i)
      8

DEBUG:asyncio:Using selector: KqueueSelector
ipdb> /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/base_events.py:654: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
  _warn(f"unclosed event loop {self!r}", ResourceWarning, source=self)
Object allocated at (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", lineno 890
    self._bootstrap_inner()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", lineno 932
    self.run()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/threading.py", lineno 870
    self._target(*self._args, **self._kwargs)
  File "/Users/mattiasa/test/ipython/env/lib/python3.8/site-packages/IPython/terminal/debugger.py", lineno 102
    line = self.pt_app.prompt()
  File "/Users/mattiasa/test/ipython/env/lib/python3.8/site-packages/prompt_toolkit/shortcuts/prompt.py", lineno 994
    return self.app.run(set_exception_handler=set_exception_handler)
  File "/Users/mattiasa/test/ipython/env/lib/python3.8/site-packages/prompt_toolkit/application/application.py", lineno 808
    loop = new_event_loop()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", lineno 758
    return get_event_loop_policy().new_event_loop()
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", lineno 656
    return self._loop_factory()
ipdb>
> /Users/mattiasa/test/ipython/test.py(6)test()
      5         import ipdb; ipdb.set_trace()
----> 6         for i in range(1, 10):
      7             self.assertEqual(i, i)

DEBUG:asyncio:Using selector: KqueueSelector
ipdb>

[mattiasa]

I'm not allowed to reopen this issue. Let me know if you want me to file a new issue instead.

[lyrixderaven]

Just to verify, I can reproduce the issue as well, at the same line in prompt_toolkit's application.py:808. Is there any quickfix to just suppress the warning for now? It's really hindering to the debugging process due to all the noise.

[craigds]

We're getting this in a (linux) docker container, in an Ubuntu 18.04 image with ipython 7.23.1 on Python 3.7.5.

Frequently appears in stderr at random:

kx[210.140069486720768]: DEBUG: asyncio selector_events(53): Using selector: EpollSelector

every couple of inputs while ipdb-ing:

/usr/lib/python3.7/asyncio/base_events.py:618: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
  source=self)
ResourceWarning: Enable tracemalloc to get the object allocation traceback

I'm not sure if this issue belongs in the ipython or ipdb repo.

[MrMino]

@craigds could you please give complete reproduction steps here? I.e. the image ID + commands?

[craigds]

it is a private build based on ubuntu 18.04. i'll try and make a minimal test case i can share

[craigds]

@MrMino I didn't get anywhere with trying to simplify our largish docker container into anything vaguely shareable.

I did try setting PYTHONTRACEMALLOC=1 and DEBUG level logging and got this:

ipdb> hellokx[53.140321883158272]: DEBUG: parso.python.diff diff(282): diff parser start
kx[53.140321883158272]: DEBUG: parso.python.diff diff(293): line_lengths old: 1; new: 1
kx[53.140321883158272]: DEBUG: parso.python.diff diff(297): -> code[replace] old[1:1] new[1:1]
kx[53.140321883158272]: DEBUG: parso.python.diff diff(425): parse_part from 1 to 1 (to 0 in part parser)
kx[53.140321883158272]: DEBUG: parso.python.diff diff(339): diff parser end
/usr/lib/python3.7/asyncio/base_events.py:618: ResourceWarning: unclosed event loop <_UnixSelectorEventLoop running=False closed=False debug=False>
  source=self)
Object allocated at (most recent call last):
  File "/usr/lib/python3.7/asyncio/events.py", lineno 660
    return self._loop_factory()

doesn't look very helpful to me, but that's all I have.

I made this happen by typing hello at the ipdb> prompt, and then pressing TAB. (There were no tab-complete results for hello but it produced the warning)

I'm happy to just filterwarnings this away in our app, but thought you might like to see this before I do.

[MrMino]

I managed to reproduce it and created a Dockerfile for it.

wget https://gist.githubusercontent.com/MrMino/a576a98f772d8c947bb67b77610dbac9/raw/a1f795d2707b97ae37db274f4e9e5eaed60623f2/Dockerfile -q -O - | \
sudo docker build --tag repro - && \
sudo docker run -it --rm repro
# Step through few lines with n and wait a second

It's still there with ipython7.24.1 on Python 3.8

I think it only happens when unittest.main() is running. Tried to reproduce this without calling it, and the warning is gone. Might be due to some interplay between prompt-toolkit and unittest.

[mzr1996]

Any updates on this problem? There are too many noisy warnings during debugging. Or any method to suppress these warnings temporarily?

[augustomen]

@mzr1996 The way to temporarily suppress warnings it to set the environment variable PYTHONWARNINGS=ignore. I had to do this to at least be able to debug with unittest. Note that this will suppress all warnings, not just this one.