RuntimeWarnings in IPython when asyncio integration is active

[iamlikeme]

Summary of problem

IPython produces RuntimeWarning on each tab-completion and enter when ddtrace's asyncio integration is active.

Which version of dd-trace-py are you using?

2.8.0

Which version of pip are you using?

24.0

Which libraries and their versions are you using?

ipython==8.23.0
ddtrace==2.8.0

`pip freeze`

asttokens==2.4.1 attrs==23.2.0 bytecode==0.15.1 cattrs==23.2.3 ddsketch==3.0.1 ddtrace==2.8.0 decorator==5.1.1 Deprecated==1.2.14 envier==0.5.1 executing==2.0.1 importlib-metadata==7.0.0 ipython==8.23.0 jedi==0.19.1 matplotlib-inline==0.1.6 opentelemetry-api==1.24.0 parso==0.8.4 pexpect==4.9.0 prompt-toolkit==3.0.43 protobuf==5.26.1 ptyprocess==0.7.0 pure-eval==0.2.2 Pygments==2.17.2 setuptools==69.2.0 six==1.16.0 sqlparse==0.4.4 stack-data==0.6.3 traitlets==5.14.2 typing_extensions==4.11.0 wcwidth==0.2.13 wrapt==1.16.0 xmltodict==0.13.0 zipp==3.18.1

How can we reproduce your problem?

In an IPython REPL:

In [1]: import ddtrace

In [2]: ddtrace.patch_all()

In [3]: # press enter
/home/piotr/repos/ddtrace-bug/.venv/lib/python3.12/site-packages/prompt_toolkit/application/application.py:1207: RuntimeWarning: coroutine 'Application.run_async.<locals>._run_async.<locals>.auto_flush_input' was never awaited
  await asyncio.wait(
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

/home/piotr/repos/ddtrace-bug/.venv/lib/python3.12/site-packages/prompt_toolkit/key_binding/key_processor.py:413: RuntimeWarning: coroutine 'KeyProcessor._start_timeout.<locals>.wait' was never awaited
  self._flush_wait_task = app.create_background_task(wait())
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

In [3]:

The problem disappears when asyncio integration is deactivated:

In [1]: import ddtrace

In [2]: ddtrace.patch_all(asyncio=False)

In [3]: # press enter

In [3]:

What is the result that you get?

I see RuntimeWarnings on every enter and tab-completion in IPython.

What is the result that you expected?

I expected to see no RuntimeWarnings :slightly_smiling_face:

[emmettbutler]

Interesting! Thanks for the report, @iamlikeme. We'll check it out.

[darkwind-noveo]

I encountered same issue when navigating with tmate in Ipython shell console

In [1]: ls
/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py:1207: RuntimeWarning: coroutine 'Application.run_async.<locals>._run_async.<locals>.auto_flush_input' was never awaited
  await asyncio.wait(
RuntimeWarning: Enable tracemalloc to get the object allocation traceback
CONTRIBUTING.md  Pipfile.lock  certificates/    scripts/    tests/
Dockerfile       README.md     docker/        locale/       pylintrc    setup.cfg   uwsgi-json.ini
Pipfile            manage.py*    pytest.ini  templates/  uwsgi.ini

/usr/local/lib/python3.11/site-packages/prompt_toolkit/key_binding/key_processor.py:413: RuntimeWarning: coroutine 'KeyProcessor._start_timeout.<locals>.wait' was never awaited
  self._flush_wait_task = app.create_background_task(wait())
RuntimeWarning: Enable tracemalloc to get the object allocation traceback

When i downgraded ddtrace from 2.8.2 to 2.7.2 it resolved my issues. (2.7.10 is good too, 2.8.0 breaks)

I tried turning on python tracemalloc stuff via env var, then written error changed itself to

In [1]: 
/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py:1207: RuntimeWarning: coroutine 'Application.run_async.<locals>._run_async.<locals>.auto_flush_input' was never awaited
  await asyncio.wait(
Object allocated at (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/application/application.py", lineno 704
    flush_task = self.create_background_task(auto_flush_input())

/usr/local/lib/python3.11/site-packages/prompt_toolkit/key_binding/key_processor.py:413: RuntimeWarning: coroutine 'KeyProcessor._start_timeout.<locals>.wait' was never awaited
  self._flush_wait_task = app.create_background_task(wait())
Object allocated at (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prompt_toolkit/key_binding/key_processor.py", lineno 413
    self._flush_wait_task = app.create_background_task(wait())

My pip freeze

aiohttp==3.9.5 aioresponses==0.7.6 aiosignal==1.3.1 aiostream==0.5.2 amqp==5.2.0 annotated-types==0.6.0 appdirs==1.4.4 asgiref==3.8.1 astroid==3.1.0 asttokens==2.4.1 async-timeout==4.0.3 asyncpg==0.29.0 attrs==23.2.0 Babel==2.14.0 bcrypt==4.1.2 beautifulsoup4==4.12.3 billiard==4.2.0 boto3==1.28.80 botocore==1.31.80 braintree==3.35.0 Brotli==1.1.0 bytecode==0.15.1 cached-property==1.5.2 cartomancer==0.1.4 cattrs==23.2.3 celery==5.3.1 certifi==2024.2.2 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 click-didyoumean==0.3.1 click-plugins==1.1.1 click-repl==0.3.0 colorama==0.4.6 coverage==6.5.0 coveralls==3.3.1 cron-descriptor==1.4.3 cryptography==42.0.5 cssselect2==0.7.0 cssutils==2.10.2 ddsketch==3.0.1 ddtrace==2.8.2 debugpy==1.8.1 decorator==5.1.1 defusedxml==0.7.1 Deprecated==1.2.14 dill==0.3.8 distlib==0.3.8 Django==4.1.13 django-add-default-value==0.10.0 django-appconf==1.0.6 django-cassette==0.8.1 django-celery-beat==2.5.0 django-chunkator==2.0.0 django-cors-headers==4.3.1 django-countries==7.6.1 django-crispy-forms==2.0 django-datadog-logger==0.7.1 django-extensions==3.2.3 django-filter==22.1 django-fsm==3.0.0 django-fsm-log==3.1.0 django-localflavor==4.0 django-model-utils==4.5.0 django-ratelimit==4.1.0 django-redis==5.4.0 django-relativedelta==2.0.0 django-request-id==1.0.0 django-ses==4.0.0 django-storages==1.14.2 django-timezone-field==6.1.0 django-tqdm==1.3.1 djangorestframework==3.14.0 djangorestframework-filters==1.0.0.dev2 docker==7.0.0 docopt==0.6.2 drf-flex-fields==1.0.2 elastic-apm==2.1.1 elasticsearch==5.5.3 elasticsearch-dsl==5.4.0 elementpath==4.4.0 envier==0.5.1 et-xmlfile==1.1.0 execnet==2.1.1 executing==2.0.1 factory-boy==3.2.1 Faker==24.11.0 filelock==3.14.0 flake8==7.0.0 flake8-tidy-imports==4.10.0 flowey==1.4.5 fonttools==4.51.0 freezegun==1.4.0 frozenlist==1.4.1 getconf==1.11.1 googleapis-common-protos==1.63.0 grpcio==1.62.2 grpcio-tools==1.62.2 html2text==2024.2.26 html5lib==1.1 idna==3.7 importlib-metadata==7.0.0 iniconfig==2.0.0 ipaddress==1.0.23 ipython==8.23.0 isodate==0.6.1 isort==5.13.2 jdcal==1.4.1 jedi==0.19.1 jmespath==1.0.1 JSON-log-formatter==1.0 jsonfield==3.1.0 junitparser==3.1.2 jwcrypto==0.8 kombu==5.3.1 lark==1.1.9 lxml==5.2.1 Markdown==3.6 MarkupSafe==2.1.5 matplotlib-inline==0.1.7 mccabe==0.7.0 mistql==0.4.11 multidict==6.0.5 oauthlib==3.2.2 openpyxl==3.0.6 opentelemetry-api==1.24.0 opentelemetry-exporter-otlp==1.24.0 opentelemetry-exporter-otlp-proto-common==1.24.0 opentelemetry-exporter-otlp-proto-grpc==1.24.0 opentelemetry-exporter-otlp-proto-http==1.24.0 opentelemetry-instrumentation==0.45b0 opentelemetry-instrumentation-asyncpg==0.45b0 opentelemetry-instrumentation-boto==0.45b0 opentelemetry-instrumentation-boto3sqs==0.45b0 opentelemetry-instrumentation-celery==0.45b0 opentelemetry-instrumentation-dbapi==0.45b0 opentelemetry-instrumentation-django==0.45b0 opentelemetry-instrumentation-elasticsearch==0.45b0 opentelemetry-instrumentation-psycopg2==0.45b0 opentelemetry-instrumentation-redis==0.45b0 opentelemetry-instrumentation-requests==0.45b0 opentelemetry-instrumentation-wsgi==0.45b0 opentelemetry-proto==1.24.0 opentelemetry-sdk==1.24.0 opentelemetry-semantic-conventions==0.45b0 opentelemetry-util-http==0.45b0 packaging==24.0 parso==0.8.4 pexpect==4.9.0 pillow==10.3.0 pipenv==2023.12.1 platformdirs==4.2.0 pluggy==1.5.0 prompt-toolkit==3.0.43 protobuf==4.25.3 psutil==5.9.8 psycopg2==2.9.9 psycopg2-binary==2.9.9 ptyprocess==0.7.0 pure-eval==0.2.2 py==1.11.0 pycodestyle==2.11.1 pycparser==2.22 pycurl==7.45.3 pydantic==2.7.0 pydantic-settings==2.2.1 pydantic_core==2.18.1 pydyf==0.9.0 pyflakes==3.2.0 Pygments==2.17.2 PyJWT==2.8.0 pylint==3.1.0 PyNaCl==1.5.0 pynliner==0.8.0 pyphen==0.14.0 pypng==0.20220715.0 pytest==7.4.4 pytest-asyncio==0.23.6 pytest-celery==1.0.0 pytest-cov==4.1.0 pytest-django==4.8.0 pytest-docker-tools==3.1.3 pytest-lazy-fixture==0.6.3 pytest-split==0.8.2 pytest-watch==4.2.0 pytest-xdist==3.3.1 python-crontab==3.0.0 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 python-stdnum==1.20 pytz==2024.1 PyYAML==6.0.1 qrcode==7.4.2 redis==5.0.3 requests==2.31.0 requests-aws4auth==1.2.3 requests-mock==1.12.1 requests-oauthlib==2.0.0 requests-toolbelt==1.0.0 responses==0.25.0 retry==0.9.2 s3transfer==0.7.0 sentry-sdk==1.45.0 sepaxml==2.1.1 simple-term-menu==1.6.4 six==1.16.0 soupsieve==2.5 spider==1.0.34 sqlparse==0.5.0 stack-data==0.6.3 suds-community==1.1.2 symba==0.36 tablib==3.6.1 text-unidecode==1.3 tinycss2==1.2.1 tokipona==1.4.0 tomlkit==0.12.4 tqdm==4.66.2 traitlets==5.14.3 typeguard==2.13.3 typelog==0.3.0 typing_extensions==4.11.0 tzdata==2024.1 unittest-xml==0.2.2 unittest-xml-reporting==3.2.0 urllib3==2.0.7 uWSGI==2.0.25.1 vine==5.1.0 virtualenv==20.26.2 watchdog==4.0.0 wcwidth==0.2.13 weasyprint==61.2 webencodings==0.5.1 Werkzeug==3.0.2 wrapt==1.16.0 xlrd==2.0.1 XlsxWriter==3.2.0 xlwt==1.3.0 xmlschema==3.3.0 xmltodict==0.13.0 yarl==1.9.4 zeep==3.4.0 zipp==3.18.1 zopfli==0.2.3

pip version 24.0

Confirming that solution DD_TRACE_ASYNCIO_ENABLED=false python3 manage.py shell helped me too.

[lvieirajr]

I have the same issues too.

[toadjaune]

Hi,

We're affected as well, and I have bisected the ddtrace lib version to to following :

• 2.7.10 is not affected
• 2.8.0 is affected

I can confirm that running my shell without asyncio integration (DD_TRACE_ASYNCIO_ENABLED=false ./manage.py shell) solves the issue.

By any chance, is there any update on this topic ? As of now, it's blocking library updates on our side.

[github-actions[bot]]

This issue has been automatically closed after a period of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error.

[toadjaune]

Hi, this is still considered blocking upgrades beyond 2.7.10 on our end.

[toadjaune]

Hi again, after further testing, I can confirm that the issue is still present on my codebase, with the latest library version (3.2.1).

Could someone please reopen this issue ?

[toadjaune]

Hi @emmettbutler , sorry for pinging you directly, could you please reopen this issue ? 🙏

[toadjaune]

Thanks ! :)

[toadjaune]

Hi, I have a new data point that may be of interest :

I just tried upgrading to 3.6.0 to see if the issue persists, and noticed a new RuntimeWarning related to the asyncio integration :

/usr/lib/python3.12/asyncio/base_events.py:1972: RuntimeWarning: coroutine 'Buffer._create_completer_coroutine.<locals>.async_completer.<locals>.refresh_while_loading' was never awaited

I'm not sure whether it's an entirely new issue, or if it's actually a new symptom of the same issue, which may further help identifying it.

[taegyunkim]

@toadjaune do you mind downloading the wheel from this link and checking whether the problem still persists? https://github.com/DataDog/dd-trace-py/actions/runs/14806384252?pr=13326 This is the same pipeline we use for building and releasing wheels to PyPI.

I was able to reproduce the problem on both macOS(arm64) and Ubuntu(x86_64) and checked that it's fixed with the PR, #13326

[toadjaune]

Hi !

I can confirm this resolves the issue and makes both errors messages disappear on both environments types that I have handy (ubuntu 24.04 server, Fedora 42 workstation)

Thank you, this is great :)

[taegyunkim]

@toadjaune Thanks for testing on your side. I'll merge and ship this PR.

[toadjaune]

Does that mean we can expect it as part of 3.7.0 ? (I'm not sure what your branching/versionning strategy on this repo is)

[taegyunkim]

@toadjaune It'll probably be picked up by 3.8.0. 3.7.0 has been cut already.

[taegyunkim]

@toadjaune It's picked up by 3.7.0rc2 https://github.com/DataDog/dd-trace-py/releases/tag/v3.7.0rc2 you'll get it sooner.

[toadjaune]

That's perfect, thanks again !