newrelic-python-agent icon indicating copy to clipboard operation
newrelic-python-agent copied to clipboard

GraphQL (graphene) transactions are slow with newrelic from v6.6.0.162 and above

Open trimailov opened this issue 2 years ago • 7 comments

Description

Upgrading anything above v6.6.0.162 will make our GraphQL endpoints slower by some margin.

From stable ~30ms to hovering between 50-100ms.

Also CPU and RAM utilization is also noticable - RAM use increased by ~20% from previous total and CPU use is additional ~5%. So for example if we used 50% RAM - we now use 60% and if we used 20% CPU - we now use 25% CPU.

I have tried to run these newrelic agent versions: v7.16.0.178, v7.14.0.177, v6.10.0.165, v6.6.0.162 - all impacted response times in same way.

I was upgrading from: v6.2.0.156 and settled at v6.4.4.161 which is latest version without graphql support (to my knowledge), but it is not impacting my applications performance.

Also - important to note that <= v6.4.4.161 reports graphql transactions as web transaction named as GraphQL endpoint view class (i.e. Django view class that serves GraphQL API), which is expected, though versions with GraphQL support (>v6.6.0.162) are reporting graphql transactions as graphene.types.resolver:dict_or_attr_resolver.

Expected Behavior

There shouldn't be noticable changes in our web servers response times and CPU/RAM utilization.

Troubleshooting or NR Diag results

Output from NR Diag. I think it's all OK, as .NET errors or Java warnings are not relevant for the project.

> ./nrdiag_x64 -c newrelic.ini

Check Results
-------------------------------------------------

Info     Base/Env/CollectEnvVars [Gathered Environment variables of current shell.]
Success  Base/Config/Collect
Success  Base/Config/Validate
Success  Base/Config/LicenseKey
Success  Base/Config/AppName
Success  Python/Config/Agent
Success  Base/Log/Copy
Info     Python/Env/Dependencies [Collected pip freeze. See pipFreeze.txt for more i...]
Info     Base/Containers/DetectDocker [Docker Daemon is Running]
Info     Base/Env/HostInfo [Collected host information]
Error    DotNetCore/Env/Versions
Success  Base/Config/ValidateLicenseKey
Info     Base/Config/RegionDetect [1 unique New Relic region(s) detected from config.]
Info     Python/Env/Version [3.9.2]
Success  Base/Collector/ConnectEU
Warning  Java/AppServer/WebSphere
Success  Base/Config/ValidateHSM
Success  Python/Requirements/Webframework

Steps to Reproduce

I will try to create minimal example application to be able to reproduce this, but it's a Django application with graphene-django used for implementing GraphQL API, with minimal djangorestframework endpoint to serve it.

Your Environment

Basic app package requirements (python 3.9.2):

graphene==2.1.8
graphene-django==2.13.0
graphql-core==2.3.2
graphql-relay==2.0.0
gunicorn==19.9.0
newrelic==7.16.0.178   # as said above behavior can be reproduce with anything from v6.6.0.162
Django==3.1.7  # I know it's old, but I was in process of upgrading packages, that's how I caught this problem
djangorestframework==3.12.1

Web app is served with gunicorn and started with newrelic-admin:

NEW_RELIC_ENVIRONMENT=$ENV NEW_RELIC_CONFIG_FILE=newrelic.ini newrelic-admin run-program gunicorn -c ./gunicorn.conf.py app.wsgi:application

To my understanding newrelic.ini is not configured beyond default, setting up API key and enabling environments for staging and production.

Additional context

Looking at newrelic's transaction details I notice that the mentioned graphene.types.resolver:dict_or_attr_resolver is called hundreds of times.

Screenshot 2022-08-16 at 19 41 22

I am not sure if this is newrelic's bug or (more likely) misimplementation of our graphene application that breaks newrelic.

trimailov avatar Aug 16 '22 16:08 trimailov

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

stale[bot] avatar Oct 15 '22 23:10 stale[bot]

Hey @trimailov; is this still an issue? Have you been able to recreate a sharable environment to validate this?

Ak-x avatar Nov 16 '22 18:11 Ak-x

It's still an issue, though have no shareable environment at hand :(

trimailov avatar Nov 17 '22 12:11 trimailov