"Error adding sample to profile (ddog_prof_Profile_add failed: Duplicate label on sample" error after enabling experimental profilers

[gibsondan]

Summary of problem

After enabling the experimental stack sampling feature for the datadog profiler (by setting DD_PROFILING_EXPORT_LIBDD_ENABLED and DD_PROFILING_STACK_V2_ENABLED to true in my process), the process repeatedly logs errors that look like this: Error adding sample to profile (ddog_prof_Profile_add failed: Duplicate label on sample: Label { key: "task id", str: None, num: 140071738420352, num_unit: None } Label { key: "task id", str: None, num: 140071738420352, num_unit: None })

I can see that there is a duplicate task id value being set, but am unsure how to prevent that from happening.

From what i can tell, the profiler is working when this is happening, but is only logging CPU information, not memory information.

Which version of dd-trace-py are you using? 2.9.5

Which version of pip are you using? 24.1.2

Which libraries and their versions are you using?

`pip freeze`

agate==1.7.1 aiohappyeyeballs==2.3.5 aiohttp==3.10.3 aiosignal==1.3.1 alembic==1.12.0 amqp==5.2.0 aniso8601==9.0.1 annotated-types==0.6.0 anyio==3.7.1 asgi-idempotency-header==0.2.0 attrs==23.2.0 Authlib==1.3.1 awswrangler==3.7.2 Babel==2.14.0 backoff==2.2.1 beautifulsoup4==4.12.3 bleach==6.1.0 boto3==1.34.72 boto3-stubs==1.34.161 botocore==1.34.72 botocore-stubs==1.34.161 bytecode==0.15.1 cachetools==5.3.3 cattrs==23.2.3 certifi==2024.2.2 cffi==1.16.0 charset-normalizer==3.3.2 click==8.1.7 colorama==0.4.6 coloredlogs==14.0 croniter==2.0.3 cryptography==42.0.5 daff==1.3.46 datadog==0.49.1 dbt-adapters==1.4.1 dbt-common==1.7.0 dbt-core==1.8.5 dbt-extractor==0.5.1 dbt-semantic-interfaces==0.5.1 ddsketch==3.0.1 ddtrace==2.9.5 deepdiff==7.0.1 defusedxml==0.7.1 Deprecated==1.2.14 docstring_parser==0.16 elastic-transport==8.15.0 elasticsearch==8.15.0 envier==0.5.1 fastapi==0.110.0 fastjsonschema==2.19.1 filelock==3.15.4 frozenlist==1.4.1 fsspec==2024.3.1 gitdb==4.0.11 github3.py==4.0.1 GitPython==3.1.42 google-api-core==2.18.0 google-auth==2.29.0 google-cloud-recaptcha-enterprise==1.19.0 googleapis-common-protos==1.63.0 gql==3.5.0 graphene==3.3 graphql-core==3.2.3 graphql-relay==3.2.0 greenlet==3.0.3 grpcio==1.62.1 grpcio-health-checking==1.62.1 grpcio-status==1.62.1 gunicorn==23.0.0 h11==0.14.0 hiredis==2.3.2 httpcore==1.0.5 httptools==0.6.1 httpx==0.27.0 humanfriendly==10.0 idna==3.7 importlib-metadata==6.11.0 isodate==0.6.1 itsdangerous==2.1.2 Jinja2==3.1.4 jmespath==1.0.1 jsonschema==4.21.1 jsonschema-specifications==2023.12.1 jupyter_client==8.6.1 jupyter_core==5.7.2 jupyterlab_pygments==0.3.0 kombu==5.2.4 kubernetes==29.0.0 leather==0.4.0 Logbook==1.5.3 lxml==4.9.4 Mako==1.3.2 markdown-it-py==3.0.0 MarkupSafe==2.1.5 mashumaro==3.12 mdurl==0.1.2 minimal-snowplow-tracker==0.0.2 mistune==3.0.2 more-itertools==10.2.0 msgpack==1.0.8 multidict==6.0.5 mypy-boto3-cloudformation==1.34.111 mypy-boto3-dynamodb==1.34.148 mypy-boto3-ec2==1.34.159 mypy-boto3-lambda==1.34.77 mypy-boto3-rds==1.34.152 mypy-boto3-s3==1.34.160 mypy-boto3-sqs==1.34.121 nbclient==0.10.0 nbconvert==7.16.3 nbformat==5.10.3 networkx==3.2.1 numpy==1.26.4 oauthlib==3.2.2 opentelemetry-api==1.23.0 ordered-set==4.1.0 orjson==3.10.0 packaging==24.0 pandas==2.2.1 pandocfilters==1.5.1 parsedatetime==2.6 passlib==1.7.4 pathspec==0.11.2 pdpyras==5.2.0 pendulum==2.1.2 pex==2.2.2 platformdirs==4.2.0 prometheus_client==0.20.0 prompt-toolkit==3.0.36 proto-plus==1.23.0 protobuf==4.25.3 psutil==6.0.0 psycopg2-binary==2.9.9 pyarrow==15.0.2 pyasn1==0.6.0 pyasn1_modules==0.4.0 pycparser==2.21 pydantic==2.6.4 pydantic_core==2.16.3 Pygments==2.17.2 PyJWT==2.8.0 PyNaCl==1.5.0 python-dateutil==2.9.0.post0 python-dotenv==1.0.1 python-gitlab==4.4.0 python-multipart==0.0.9 python-slugify==8.0.4 python3-saml==1.16.0 pytimeparse==1.1.8 pytz==2024.1 pytzdata==2020.1 PyYAML==6.0.1 pyzmq==25.1.2 questionary==2.0.1 redis==5.0.3 referencing==0.34.0 requests==2.32.3 requests-oauthlib==2.0.0 requests-toolbelt==1.0.0 rich==13.7.1 rpds-py==0.18.0 rsa==4.9 s3transfer==0.10.1 segment-analytics-python==2.3.2 shellingham==1.5.4 six==1.16.0 slack-bolt==1.18.1 slack_sdk==3.27.1 smmap==5.0.1 sniffio==1.3.1 soupsieve==2.5 SQLAlchemy==1.4.52 sqlglot==23.2.0 sqlglotrs==0.1.3 sqlparse==0.5.1 starlette==0.36.3 stripe==3.5.0 structlog==24.1.0 tabulate==0.9.0 text-unidecode==1.3 tinycss2==1.2.1 tomli==2.0.1 toposort==1.10 tornado==6.4 tqdm==4.66.5 traitlets==5.14.2 typer==0.11.0 types-awscrt==0.21.2 types-boto==2.49.18.20240205 types-pyOpenSSL==24.0.0.20240311 types-redis==4.6.0.20240311 types-s3transfer==0.10.1 types-stripe==3.5.2.20240106 typing_extensions==4.10.0 tzdata==2024.1 ulid-py==1.1.0 universal_pathlib==0.2.2 uritemplate==4.1.1 urllib3==2.2.2 uv==0.2.36 uvicorn==0.29.0 uvloop==0.19.0 validators==0.23.2 vine==5.1.0 watchdog==4.0.0 watchfiles==0.21.0 wcwidth==0.2.13 webencodings==0.5.1 websocket-client==1.7.0 websockets==10.4 wrapt==1.16.0 xmlsec==1.3.13 xmltodict==0.13.0 yarl==1.9.4 zipp==3.18.1

How can we reproduce your problem?

What is the result that you get?

Logspew like the following, no memory profiling

Error adding sample to profile (ddog_prof_Profile_add failed: Duplicate label on sample: Label { key: "task id", str: None, num: 140071724400704, num_unit: None } Label { key: "task id", str: None, num: 140071724400704, num_unit: None })
Error adding sample to profile (ddog_prof_Profile_add failed: Duplicate label on sample: Label { key: "task id", str: None, num: 140071724400704, num_unit: None } Label { key: "task id", str: None, num: 140071724400704, num_unit: None })
Error adding sample to profile (ddog_prof_Profile_add failed: Duplicate label on sample: Label { key: "task id", str: None, num: 140072623446848, num_unit: None } Label { key: "task id", str: None, num: 140072623446848, num_unit: None })
Error adding sample to profile (ddog_prof_Profile_add failed: Duplicate label on sample: Label { key: "task id", str: None, num: 140072623446848, num_unit: None } Label { key: "task id", str: None, num: 140072623446848, num_unit: None })
Error adding sample to profile (ddog_prof_Profile_add failed: Duplicate label on sample: Label { key: "task id", str: None, num: 140072623443968, num_unit: None } Label { key: "task id", str: None, num: 140072623443968, num_unit: None })

What is the result that you expected?

No logspew, memory profiling

[gibsondan]

A couple other details in case they're helpful: This is a starlette + uvicorn app running a graphql server. We've observed in in two different services with a similar setup but different business logic and schemas, after applying the experimental stack sampling.

[github-actions[bot]]

This issue has been automatically closed after a period of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error.