dd-trace-php icon indicating copy to clipboard operation
dd-trace-php copied to clipboard

[Bug]: Spike memory consumption since 1.12.0?

Open AntoineLemaire opened this issue 3 months ago • 3 comments

Bug report

Hello, we deployed yesterday a new release of our application where DDTrace 1.12.0 was included. And it result with some memory consumption spikes we didn't had before. Due to that, it trigger our autoscalling and deploy time to time more pods than usual. Our application is running on Symfony 4.4 and PHP 7.4.33 docker images, managed with k8s.

I reverted back to 1.11.0 (but with same application release) and the issue disappeared

Image

I don't really know what I could share to debug this issue.

PHP version

7.4.33

Tracer or profiler version

1.12.0

Installed extensions

[PHP Modules]
apcu
bcmath
Core
ctype
curl
date
ddappsec
ddtrace
dom
exif
fileinfo
filter
ftp
gd
hash
iconv
intl
json
libxml
mbstring
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
redis
Reflection
session
SimpleXML
soap
sockets
sodium
SPL
sqlite3
standard
tokenizer
xml
xmlreader
xmlwriter
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
ddappsec
ddtrace

Output of phpinfo()

{
    "date":"2025-09-04T12:02:09Z",
    "os_name":"Linux yp-main-5875f48fd5-47smm 6.1.134-152.225.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Wed May 7 09:10:59 UTC 2025 x86_64",
    "os_version":"6.1.134-152.225.amzn2023.x86_64",
    "version":"1.12.0",
    "lang":"php",
    "lang_version":"7.4.33",
    "env":"prod-singapore",
    "enabled":true,
    "service":"yp-main",
    "enabled_cli":true,
    "agent_url":"http:\/\/10.11.45.233:8126",
    "debug":false,
    "analytics_enabled":false,
    "sample_rate":-1,
    "sampling_rules":[
        {
            "service":"yp-main",
            "sample_rate":0.1
        }
    ],
    "tags":[],
    "service_mapping":[],
    "distributed_tracing_enabled":true,
    "dd_version":"tmp-php_debug_e42ce3eba7_Dmaster_aaa2e837",
    "architecture":"x86_64",
    "instrumentation_telemetry_enabled":true,
    "sapi":"fpm-fcgi",
    "datadog.trace.sources_path":"\/opt\/datadog\/dd-library\/1.12.0\/dd-trace-sources\/src",
    "open_basedir_configured":false,
    "uri_fragment_regex":null,
    "uri_mapping_incoming":null,
    "uri_mapping_outgoing":null,
    "auto_flush_enabled":false,
    "generate_root_span":true,
    "http_client_split_by_domain":false,
    "measure_compile_time":true,
    "report_hostname_on_root_span":false,
    "traced_internal_functions":null,
    "enabled_from_env":true,
    "opcache.file_cache":null,
    "sidecar_trace_sender":false,
    "loaded_by_ssi":false
}

Upgrading from

1.11.0

AntoineLemaire avatar Sep 04 '25 12:09 AntoineLemaire

Hey @AntoineLemaire, thanks for reaching out!

How fast does the memory spike in relation to the amount of requests? Can you observe whether it's the fpm processes or the datadog-ipc-helper process whose memory consumption spikes? If neither, do the files in /dev/shm grow a lot in size?

bwoebi avatar Sep 04 '25 16:09 bwoebi

Hey, as far as we see, it was only fpm.

Image

AntoineLemaire avatar Sep 08 '25 07:09 AntoineLemaire

Adding some brief additional anecdotes, I will elaborate more when able.

We are seeing php8.4-fpm processes kept busy, to the point of server inoperability, when using PHP 8.4.12 combined with ddtrace 1.12. This does not occur with PHP 8.4.11 and ddtrace 1.11.

We are using Symfony version 7.3 on Debian 12.12.

When disabling ddtrace 1.12, server behaves normally. Haven't been able to pinpoint contributing factors yet.

inimicus avatar Sep 13 '25 17:09 inimicus