[BUG] Upgrading from 0.65.1 to 0.68.1 causes segfault in cli scripts

[mente]

Bug description

We have an automated dependabot running a routine update of different libraries and extensions. The update of dd-trace-php extension (and library) from 0.65.1 to 0.68.1 broke our deployment pipeline because cli started segfaulting.

I've did some experimentation back and forth and discovered that removing env variable DD_TRACE_ANALYTICS_ENABLED solves the segfault issue. While this is an outdated setting, I don't think it should trigger segfault.

PHP version

$ php -v
PHP 7.4.27 (cli) (built: Dec 21 2021 21:36:53) ( NTS )
Copyright (c) The PHP Group
Zend Engine v3.4.0, Copyright (c) Zend Technologies
    with Zend OPcache v7.4.27, Copyright (c), by Zend Technologies
    with ddtrace v0.68.1, Copyright Datadog, by Datadog

Tracer version

 php -r "echo phpversion('ddtrace').PHP_EOL;"
0.68.1

Installed extensions

$ php -m
[PHP Modules]
apcu
bcmath
Core
ctype
curl
date
ddtrace
dom
fileinfo
filter
ftp
gd
hash
iconv
igbinary
intl
json
libxml
mbstring
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
pdo_sqlite
Phar
posix
readline
Reflection
session
SimpleXML
sockets
sodium
SPL
sqlite3
standard
tokenizer
xml
xmlreader
xmlwriter
Zend OPcache
zip
zlib

[Zend Modules]
Zend OPcache
ddtrace

OS info

We're using stock PHP docker images, the one in question is derived from php:7.4.27-fpm which uses debian:bullseye.

$ cat /etc/os-release | grep -E "(NAME)|(VERSION)"
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye

Diagnostics and configuration

I don't have a web access to the script, so can't shot it here

Upgrading info

We've been successfully using 0.65.1 before, upgrade to 0.68.1 broke prod.

[krakjoe]

Is there any chance you can get a backtrace (help here)?

[mente]

I can try but never did it in dockerised environment. Will be back later

[mente]

I'm sorry, it's taking of me more than I can anticipate. But I've narrowed down the problem to easier reproduce.

Please consider following dockerfile:

FROM php:7.4.27-fpm as base
ARG DATADOG_VERSION=0.68.1

# Install DataDog PHP extension
RUN curl -s -L --fail https://github.com/DataDog/dd-trace-php/releases/download/${DATADOG_VERSION}/datadog-php-tracer_${DATADOG_VERSION}_amd64.deb --output datadog-php-tracer.deb \
        && dpkg -i datadog-php-tracer.deb \
        && rm ./datadog-php-tracer.deb \
        && echo "DataDog installed at version ${DATADOG_VERSION}"

and following steps to build and run it:

$ docker build -t datadog-reproduce .
[+] Building 8.9s (6/6) FINISHED
...

$ docker run --rm -it datadog-reproduce bash
root@a2a39dd95cec:/var/www/html# echo '<?php echo "Hello World";' > /tmp/hello.php
#run without any env variables
root@a2a39dd95cec:/var/www/html# php /tmp/hello.php
Hello Worldroot@a2a39dd95cec:/var/www/html#
#enable tracing for CLI and old env variable
root@a2a39dd95cec:/var/www/html# DD_TRACE_ANALYTICS_ENABLED=true DD_TRACE_CLI_ENABLED=true !!
DD_TRACE_ANALYTICS_ENABLED=true DD_TRACE_CLI_ENABLED=true php /tmp/hello.php
Hello WorldSegmentation fault

[bwoebi]

Thanks a lot for the reproducer, we plan to release the fix for this soon as a patch version release.

[SammyK]

@mente The fix for this has been included in the latest release of the tracer. Please let us know if you run into any more issues with it. Thank you again for the report and the repro!

[mente]

Happy to help, thanks for update!

[tsalinguepumpkin]

Still having some issue on 0.68.2

bash-4.4# php -d memory_limit=-1 -d display_errors=on -r "if (hash('SHA384', file_get_contents('/tmp/composer-setup.php')) !== trim(file_get_contents('/tmp/composer-setup.sig'))) { unlink('/tmp/composer-setup.php'); echo 'Invalid installer' . PHP_EOL; exit(1); }"
Segmentation fault

If I remove the tracer it works fine

bash-4.4# apk del datadog-php-tracer
(1/2) Purging datadog-php-tracer (0.68.2)
(2/2) Purging libexecinfo (1.1-r0)
OK: 358 MiB in 130 packages
bash-4.4# php /tmp/composer-setup.php --version=1.10.15 --no-ansi --install-dir=/usr/local/bin --filename=composer
PHP Warning:  PHP Startup: Unable to load dynamic library '/opt/datadog-php/extensions/ddtrace-20180731-alpine.so' (tried: /opt/datadog-php/extensions/ddtrace-20180731-alpine.so (Error loading shared library /opt/datadog-php/extensions/ddtrace-20180731-alpine.so: No such file or directory), /usr/lib/php/7.3/modules//opt/datadog-php/extensions/ddtrace-20180731-alpine.so.so (Error loading shared library /usr/lib/php/7.3/modules//opt/datadog-php/extensions/ddtrace-20180731-alpine.so.so: No such file or directory)) in Unknown on line 0
All settings correct for using Composer
Downloading...

Composer (version 1.10.15) successfully installed to: /usr/local/bin/composer
Use it: php /usr/local/bin/composer

@SammyK @bwoebi How can we reopen this ? Also I think this create a lot of coredump in our worker server that is using supervisord to launch php script.

[SammyK]

@tsalinguepumpkin Thank you for the extra info provided over Slack today. I was unable to reproduce this issue, but the team will continue to investigate this issue and update this thread when we have more information. Have a nice weekend, and thank you for the report!

[bwoebi]

@tsalinguepumpkin You mention you have core dumps - is it possible for you to share the stacktrace? (See also https://docs.datadoghq.com/tracing/setup_overview/setup/php/?tab=containers#install-debug-symbols on how to dump the stacktrace from a core dump)

[labbati]

@tsalinguepumpkin did you have a chance to open a ticket and attach a core dump? We'd find it incredible useful to investigate it. Thank you

[tsalinguepumpkin]

@labbati I did , no news from the support yet