[BUG] DDtrace extension triggering SIGSEGV/segfaults during exception handler

[silvervest]

Bug description

Based on the core dump and gdb's zbacktrace, this is occurring in a zend call to array_reverse via the bridge/_generated_tracer.php script. Not sure how it's making zend engine crash but here we are! For reference, line 885 in _generated_tracer.php is https://github.com/DataDog/dd-trace-php/blob/master/src/DDTrace/ScopeManager.php#L135

I have snipped the following for minor opsec, but I'm happy to provide the core dump securely to a DD dev if needed for debugging etc

Reading symbols from /usr/local/sbin/php-fpm...done.
[New LWP 45]
[New LWP 120]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm:'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x000055ef155c927f in zend_mm_alloc_small (heap=0x7f256aa00040, bin_num=9, __zend_filename=0x55ef16158b57 "/usr/src/php/Zend/zend_hash.c", __zend_lineno=279, __zend_orig_filename=0x0,
    __zend_orig_lineno=0) at /usr/src/php/Zend/zend_alloc.c:1256
1256			heap->free_slot[bin_num] = p->next_free_slot;
[Current thread is 1 (Thread 0x7f256ad71640 (LWP 45))]
(gdb) zbacktrace
[0x7f256aa15860] array_reverse(array(0)[0x7f256aa158b0]) [internal function]
[0x7f256aa15720] DDTrace\ScopeManager->reconcileInternalAndUserland() /opt/datadog/dd-library/0.76.2/dd-trace-sources/bridge/_generated_tracer.php:885
[0x7f256aa15680] DDTrace\ScopeManager->getActive() /opt/datadog/dd-library/0.76.2/dd-trace-sources/bridge/_generated_tracer.php:822
[0x7f256aa15610] DDTrace\Tracer->getActiveSpan() /opt/datadog/dd-library/0.76.2/dd-trace-sources/bridge/_generated_tracer.php:1069
[0x7f256aa15590] {closure}(array(7)[0x7f256aa155e0]) /[snip]/bootstrap.php:78
[0x7f256aa15520] call_user_func(object[0x7f256aa15570], array(7)[0x7f256aa15580]) [internal function]
[0x7f256aa15400] Monolog\Logger->addRecord(400, "Warning - Undefined property: [snip]::$snip in /[snip]/model.php on line 343", array(1)[0x7f256aa15470]) /[snip]/monolog/monolog/src/Monolog/Logger.php:340
[0x7f256aa15370] Monolog\Logger->log(400, "Warning - Undefined property: [snip]::$snip in /[snip]/model.php on line 343", array(1)[0x7f256aa153e0]) /[snip]/monolog/monolog/src/Monolog/Logger.php:614
[0x7f256aa152e0] [snip]\Log->log(400, "Warning - Undefined property: [snip]::$snip in /[snip]/model.php on line 343", array(1)[0x7f256aa15350]) /[snip]/log.php:203
[0x7f256aa15230] [snip]\Log->write(400, "Warning - Undefined property: [snip]::$snip in /[snip]/model.php on line 343", array(1)[0x7f256aa152a0]) /[snip]/log.php:230
[0x7f256aa151a0] logger(400, "Warning - Undefined property: [snip]::$snip in /[snip]/model.php on line 343", array(1)[0x7f256aa15210]) /[snip]/base.php:73
[0x7f256aa15100] [snip]\Errorhandler->exception_handler(object[0x7f256aa15150]) /[snip]/errorhandler.php:128
[0x7f256aa15080] {closure}(object[0x7f256aa150d0]) /[snip]/bootstrap.php:96

PHP version

# php -v
PHP Warning:  PHP Startup: Unable to load dynamic library 'ddappsec.so' (tried: /usr/local/lib/php/extensions/debug-non-zts-20210902/ddappsec.so (/usr/local/lib/php/extensions/debug-non-zts-20210902/ddappsec.so: cannot open shared object file: No such file or directory), /usr/local/lib/php/extensions/debug-non-zts-20210902/ddappsec.so.so (/usr/local/lib/php/extensions/debug-non-zts-20210902/ddappsec.so.so: cannot open shared object file: No such file or directory)) in Unknown on line 0
PHP 8.1.8 (cli) (built: Aug  2 2022 00:39:35) (NTS DEBUG)
Copyright (c) The PHP Group
Zend Engine v4.1.8, Copyright (c) Zend Technologies
    with Zend OPcache v8.1.8, Copyright (c), by Zend Technologies
    with ddtrace v0.76.2, Copyright Datadog, by Datadog

[silvervest]

I forgot to note that I haven't been able to repro this SIGSEGV if I don't have the dd-trace-php extension installed which is why I'm bringing it here :)

[silvervest]

From further testing and experimenting I can only reproduce this when both dd-trace and PHP's JIT are enabled - if either are disabled, it seems stable. I have mucked about changing memory assigned to JIT buffer and that doesn't seem to make a difference.

[bwoebi]

Hey there,

When things are failing in zend_mm_alloc* it's mostly prior corruption. May I ask you to run this script once with valgrind? I.e. USE_ZEND_ALLOC=0 valgrind --trace-children=yes -- php-fpm -F --fpm-config <CONFIG_FILE_PATH> <MORE_OPTIONS> should do the job.

Thanks!

[silvervest]

Sorry for the delay, work work work. I've jumped back into this today and cannot get it to reproduce with or without valgrind, or with or without the dd trace ext or JIT so I don't even know where to go from here.

[bwoebi]

@silvervest So, it was an one-off error, or is the issue that you cannot reproduce it reliably enough to provide us with more information? (but it happens regularly?)

[silvervest]

@bwoebi it was happening regularly when I reported it, but no longer.

I can't ascertain any specific change (our code, environment, or change in dd-trace) that could have caused it originally or resolved, but I guess just close this for now and I can reopen if I get a reoccurrence and am able to get better debug info?

[bwoebi]

Okay, then let's close it for now :-)