newrelic-python-agent
newrelic-python-agent copied to clipboard
newrelic
newrelic-7.14.0.177 asyncio.exceptions:CancelledError
[NOTE]: # Server stopped working
Description After upgrading from newrelic-7.10.0.175 to newrelic-7.14.0.177
Expected Behavior Nothing should happen in terms of the Application performance
Troubleshooting or NR Diag results Stack trace Traceback(most recent call last): File "/app/.heroku/python/lib/python3.9/site-packages/sentry_sdk/integrations/asgi.py", line 159, in _run_app File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/api/asgi_application.py", line 359, in nr_async_asgi File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/common/async_proxy.py", line 124, in throw File "/app/.heroku/python/lib/python3.9/site-packages/newrelic/api/asgi_application.py", line 69, in call File "/app/.heroku/python/lib/python3.9/site-packages/django/core/handlers/asgi.py", line 154, in call File "/app/.heroku/python/lib/python3.9/site-packages/asgiref/sync.py", line 435, in call File "/app/.heroku/python/lib/python3.9/asyncio/tasks.py", line 442, in wait_for
Steps to Reproduce TBD. We are using Daphne/Websockets Django app on Heroku
Your Environment Python python-3.9.12
pip list
Package Version
----------------------- -----------
aioredis 1.3.1
amqp 5.1.1
asgiref 3.5.2
async-timeout 4.0.2
attrs 21.4.0
autobahn 22.4.2
Automat 20.2.0
bcrypt 3.2.2
beautifulsoup4 4.11.1
billiard 3.6.4.0
boto3 1.23.4
botocore 1.26.5
celery 5.2.6
certifi 2022.5.18.1
cffi 1.15.0
cfgv 3.3.1
channels 3.0.4
channels-redis 3.4.0
charset-normalizer 2.0.12
click 8.1.3
click-didyoumean 0.3.0
click-plugins 1.1.1
click-repl 0.2.0
constantly 15.1.0
cryptography 37.0.2
daphne 3.0.2
debugpy 1.6.0
defusedxml 0.7.1
Deprecated 1.2.13
distlib 0.3.4
Django 3.2.13
django-appconf 1.0.5
django-cors-headers 3.12.0
django-debug-toolbar 3.4.0
django-environ 0.8.1
django-extensions 3.1.5
django-filter 21.1
django-hosts 5.1
django-log-request-id 2.0.0
django-memoize 2.3.1
django-multiselectfield 0.1.12
django-pglocks 1.0.4
django-querycount 0.7.0
django-redis 5.2.0
django-select2 7.10.0
django-spurl 0.6.8
django-storages 1.12.3
django-timezone-field 4.2.3
django-webpack-loader 1.5.0
djangorestframework 3.13.1
dsnparse 0.1.15
factory-boy 3.2.1
Faker 13.11.1
filelock 3.7.0
fuzzywuzzy 0.18.0
hiredis 2.0.0
html5lib 1.1
hyperlink 21.0.0
identify 2.5.1
idna 3.3
incremental 21.3.0
iniconfig 1.1.1
isodate 0.6.1
jmespath 1.0.0
kombu 5.2.4
lxml 4.7.0
msgpack 1.0.3
nanoid 2.0.0
newrelic 7.10.0.175
nodeenv 1.6.0
oauthlib 3.2.0
packaging 21.3
paramiko 2.11.0
phonenumbers 8.12.48
Pillow 9.1.1
pip 22.1.2
platformdirs 2.5.2
pluggy 1.0.0
pre-commit 2.19.0
prompt-toolkit 3.0.29
psycopg2-binary 2.9.3
py 1.11.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.21
Pygments 2.7.3
PyJWT 2.4.0
PyNaCl 1.5.0
pyOpenSSL 22.0.0
pyparsing 3.0.9
pytest 7.1.2
pytest-asyncio 0.18.3
pytest-django 4.5.2
python-dateutil 2.8.2
python-Levenshtein 0.12.2
python-social-auth 0.3.6
python3-openid 3.2.0
python3-saml 1.14.0
pytz 2022.1
PyYAML 6.0
redis 4.3.1
requests 2.27.1
requests-oauthlib 1.3.1
s3transfer 0.5.2
sentry-sdk 1.5.12
service-identity 21.1.0
setuptools 60.10.0
six 1.16.0
slugid 2.0.0
social-auth-app-django 5.0.0
social-auth-core 4.2.0
soupsieve 2.3.2.post1
sqlparse 0.4.2
toml 0.10.2
tomli 2.0.1
tornado 6.1
twilio 7.9.1
Twisted 22.4.0
txaio 22.2.1
typing_extensions 4.2.0
urllib3 1.26.9
URLObject 2.4.3
vine 5.0.0
virtualenv 20.14.1
waiting 1.4.1
wcwidth 0.2.5
webencodings 0.5.1
wheel 0.37.1
whitenoise 6.1.0
wrapt 1.14.1
XlsxWriter 3.0.3
xmlformatter 0.2.4
xmlsec 1.3.12
zcrmsdk 2.0.13
zope.interface 5.4.0
Same here, after upgrading to 7.14 or above, our Daphne+Django+Channels async http service started to timeout requests with "WebRequest ... took too long to shut down and was killed" errors.
Python: 3.8.12 packages (with NewRelic now pinned to 7.12):
aioredis 1.3.1
amqp 5.1.1
asgiref 3.5.2
async-timeout 4.0.2
attrs 22.1.0
autobahn 22.7.1
Automat 20.2.0
billiard 3.6.4.0
celery 5.1.2
certifi 2021.5.30
cffi 1.15.1
channels 3.0.4
channels-redis 3.3.1
charset-normalizer 2.1.1
click 7.1.2
click-didyoumean 0.3.0
click-plugins 1.1.1
click-repl 0.2.0
constantly 15.1.0
coverage 5.5
cryptography 38.0.1
daphne 3.0.2
decorator 5.1.1
defusedxml 0.7.1
Django 3.2.6
django-allauth 0.45.0
django-dotenv 1.4.2
django-webpack-loader 0.7.0
djangorestframework 3.12.1
durabledict 0.9.4
hiredis 2.0.0
hyperlink 21.0.0
idna 3.3
incremental 21.3.0
iniconfig 1.1.1
jsonpath-rw 1.4.0
jsonpickle 2.2.0
kombu 5.2.1
MarkupSafe 2.1.1
more-itertools 8.14.0
msgpack 1.0.4
newrelic 7.12.0.176
oauthlib 3.2.0
packaging 21.3
pip 22.2.2
pluggy 0.13.1
ply 3.11
prompt-toolkit 3.0.31
psycopg2-binary 2.9.1
py 1.11.0
pyasn1 0.4.8
pyasn1-modules 0.2.8
pycparser 2.21
pydantic 1.8.2
PyJWT 2.3.0
pyOpenSSL 22.0.0
pyparsing 3.0.9
pytest 6.0.1
pytest-cov 3.0.0
pytest-django 4.5.2
pytest-env 0.6.2
python3-openid 3.2.0
pytz 2022.2.1
PyYAML 5.3.1
redis 3.5.3
requests 2.28.1
requests-oauthlib 1.3.1
sentry-sdk 1.3.1
service-identity 21.1.0
setuptools 57.5.0
six 1.16.0
slack-sdk 3.3.0
sqlparse 0.4.2
toml 0.10.2
Twisted 22.8.0
txaio 22.2.1
typing_extensions 4.3.0
urllib3 1.26.12
vine 5.0.0
wcwidth 0.2.5
websockets 10.1
Werkzeug 2.2.2
wheel 0.37.0
zope.interface 5.4.0
This is still happening today on the latest version of newrelic Python agent.
Essentially as soon as I send a single web-sockets message using Django channels, the server not only times out that request but fails all future requests. I am running Django with newrelic-admin run-program gunicorn mysite.asgi --pythonpath mysite --timeout 15 --max-requests 10000 --access-logformat EMPTY_ACCESS_LOG --worker-class uvicorn.workers.UvicornWorker
This issue happens both on Heroku and also in localhost.
Downgrading to newrelic 7.10 fixes it.
We were able to reproduce this issue. It appears to be related to aioredis instrumentation. Specifically, channels-redis<4 used aioredis to implement the redis connection for the channel layer. Newer versions (channels-redis>=4) seem to be unaffected, as the implementation of redis changed to the async client from the redis package instead of aioredis. (Which doesn't use the same instrumentation.)
From our investigation we believe the underlying issue was fixed in v8.2.1, with a fix for an additional crash in v8.3.0.
Can you all try the latest version of the agent (v8.4.0+) and see if the issue is fixed for you?
