[BUG]: Telemetry dependency collection massively consumes CPU at 1 minute mark impacting our production services

[kravnaut]

Tracer Version(s)

2.19.3

Python Version(s)

3.12.8

Pip Version(s)

24.3.1

Bug Report

Recently (or at least in this version we just upgraded to) added feature gated by DD_TELEMETRY_DEPENDENCY_COLLECTION_ENABLED seems to be

We have profiled the application and upon learning it was this feature we saw in your code it was gated using env var DD_TELEMETRY_DEPENDENCY_COLLECTION_ENABLED which we have since engaged to disable it. This resolved the problem for us.

Reproduction Code

CPU spikes to 100% and stays there for 5+ seconds impacting ongoing request handling in the container.

Error Logs

No response

Libraries in Use

No response

Operating System

No response

[kravnaut]

Here is austin profiler snapshot of the process running this code in ddtrace's thread with -s setting on to only measure non-idle frames

The heaviest path:

get_module_distribution_versions (/usr/local/lib/python3.12/site-packages/ddtrace/internal/packages.py:77)
> update_imported_dependencies (/usr/local/lib/python3.12/site-packages/ddtrace/internal/telemetry/data.py:76)
> TelemetryWriter._app_dependencies_loaded_event (/usr/local/lib/python3.12/site-packages/ddtrace/internal/telemetry/writer.py:419)
> TelemetryWriter.periodic (/usr/local/lib/python3.12/site-packages/ddtrace/internal/telemetry/writer.py:627)

[P403n1x87]

@kravnaut thanks for reporting this. We have merged a fix for this recently (#12327) and it should become available with the next patch releases.

[joshfermin]

@P403n1x87 was this part of the 2.21.2 release? we are still seeing significant CPU performance issues as well on that version.

[diranged]

We just ran into this also - both on the 2.21.4 release as well as the 3.3.1 release... thanks for putting up this issue, setting DD_TELEMETRY_DEPENDENCY_COLLECTION_ENABLED=false seems to have resolved the issue for us!

[taegyunkim]

I was also wondering whether there would be a way to read metadata files only once and never again, i.e. don't call importlib.metadata again.

[github-actions[bot]]

This issue has been automatically closed after a period of inactivity. If it's a feature request, it has been added to the maintainers' internal backlog and will be included in an upcoming round of feature prioritization. Please comment or reopen if you think this issue was closed in error.