dd-trace-php
dd-trace-php copied to clipboard
DataDog
[Bug]: SIGSEGV of coms-writer
Bug report
coms-writer causes SIGSEGV when exiting. It happens once every few days.
The backtrace of apache2+php is as follows:
$ gdb /usr/sbin/apache2 --core ./CoreDump
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.2) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...
Reading symbols from /usr/lib/debug/.build-id/b9/d8d808bc1ceea80f6e55b07163298a6eb44c89.debug...
[New LWP 966983]
[New LWP 966752]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f5153905700 (LWP 966983))]
(gdb) backtrace
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f5182f7e859 in __GI_abort () at abort.c:79
#2 0x00007f5182fe926e in __libc_message (action=action@entry=do_abort, fmt=0x7f5153904cf0 "", fmt@entry=0x7f5183113298 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#3 0x00007f5182ff12fc in malloc_printerr (str=str@entry=0x7f5183115670 "double free or corruption (out)") at malloc.c:5347
#4 0x00007f5182ff2fa0 in _int_free (av=0x7f5183148b80 <main_arena>, p=0x7f514c02e660, have_lock=<optimized out>, have_lock@entry=0) at malloc.c:4314
#5 0x00007f5182ff688d in __GI___libc_free (mem=0x7f514c02e670) at malloc.c:3125
#6 tcache_thread_shutdown () at malloc.c:2964
#7 __malloc_arena_thread_freeres () at arena.c:951
#8 0x00007f5182ffb0a0 in __libc_thread_freeres () at thread-freeres.c:38
#9 0x00007f518315662f in start_thread (arg=<optimized out>) at pthread_create.c:491
#10 0x00007f518307b353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) thread apply all bt
Thread 2 (Thread 0x7f5182ee1380 (LWP 966752)):
#0 __pthread_clockjoin_ex (threadid=139987271046912, thread_return=thread_return@entry=0x0, clockid=clockid@entry=0, abstime=abstime@entry=0x0, block=block@entry=true) at pthread_join_common.c:145
#1 0x00007f51831579f3 in __pthread_join (threadid=<optimized out>, thread_return=thread_return@entry=0x0) at pthread_join.c:24
#2 0x00007f517bdfb52a in ddtrace_coms_flush_shutdown_writer_synchronous () at /home/circleci/datadog/tmp/build_extension/ext/coms.c:1361
#3 0x00007f517bdf09fa in zm_shutdown_ddtrace (type=<optimized out>, module_number=62) at /home/circleci/datadog/tmp/build_extension/ext/ddtrace.c:1493
#4 0x00007f51820d4f9f in module_destructor (module=module@entry=0x559d77deb0d0) at ./Zend/zend_API.c:2563
#5 0x00007f51820cebd0 in module_destructor_zval (zv=<optimized out>) at ./Zend/zend.c:761
#6 0x00007f51820e07b9 in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=61, ht=<optimized out>) at ./Zend/zend_hash.c:1305
#7 _zend_hash_del_el (p=0x559d77b599b0, idx=61, ht=0x7f51822bd0a0 <module_registry>) at ./Zend/zend_hash.c:1328
#8 zend_hash_graceful_reverse_destroy (ht=ht@entry=0x7f51822bd0a0 <module_registry>) at ./Zend/zend_hash.c:1782
#9 0x00007f51820d3830 in zend_destroy_modules () at ./Zend/zend_API.c:1995
#10 0x00007f51820cfb67 in zend_shutdown () at ./Zend/zend.c:1048
#11 0x00007f518206f7c2 in php_module_shutdown () at ./main/main.c:2478
#12 php_module_shutdown () at ./main/main.c:2455
#13 0x00007f518206f89d in php_module_shutdown_wrapper (sapi_globals=<optimized out>) at ./main/main.c:2448
#14 0x00007f5182159d65 in php_apache_child_shutdown (tmp=<optimized out>) at ./sapi/apache2handler/sapi_apache2.c:422
#15 0x00007f518318fd2e in apr_pool_destroy () from /lib/x86_64-linux-gnu/libapr-1.so.0
#16 0x00007f5182e00303 in clean_child_exit (code=code@entry=0) at prefork.c:227
#17 0x00007f5182e00904 in child_main (child_num_arg=child_num_arg@entry=20, child_bucket=child_bucket@entry=0) at prefork.c:636
#18 0x00007f5182e00c9a in make_child (s=0x7f5182ebb4a0, slot=slot@entry=20) at prefork.c:717
#19 0x00007f5182e01905 in prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:975
#20 0x0000559d76107488 in ap_run_mpm (pconf=0x7f518324c028, plog=0x7f5182eb4028, s=0x7f5182ebb4a0) at mpm_common.c:94
#21 0x0000559d760ff499 in main (argc=<optimized out>, argv=<optimized out>) at main.c:819
Thread 1 (Thread 0x7f5153905700 (LWP 966983)):
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f5182f7e859 in __GI_abort () at abort.c:79
#2 0x00007f5182fe926e in __libc_message (action=action@entry=do_abort, fmt=0x7f5153904cf0 "", fmt@entry=0x7f5183113298 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#3 0x00007f5182ff12fc in malloc_printerr (str=str@entry=0x7f5183115670 "double free or corruption (out)") at malloc.c:5347
#4 0x00007f5182ff2fa0 in _int_free (av=0x7f5183148b80 <main_arena>, p=0x7f514c02e660, have_lock=<optimized out>, have_lock@entry=0) at malloc.c:4314
#5 0x00007f5182ff688d in __GI___libc_free (mem=0x7f514c02e670) at malloc.c:3125
#6 tcache_thread_shutdown () at malloc.c:2964
#7 __malloc_arena_thread_freeres () at arena.c:951
#8 0x00007f5182ffb0a0 in __libc_thread_freeres () at thread-freeres.c:38
#9 0x00007f518315662f in start_thread (arg=<optimized out>) at pthread_create.c:491
#10 0x00007f518307b353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
The backtrace of php-cli is as follows:
$ gdb /usr/bin/php7.4 --core ./CoreDump
GNU gdb (Ubuntu 9.2-0ubuntu1~20.04.2) 9.2
Copyright (C) 2020 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/bin/php7.4...
Reading symbols from /usr/lib/debug/.build-id/e9/d2aac990b62feff9dfdee5cbc8756b082940bf.debug...
warning: core file may not match specified executable file.
[New LWP 966793]
[New LWP 966777]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/bin/php /home/gree/src/cli/gree_async.php --restart --all'.
Program terminated with signal SIGABRT, Aborted.
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
50 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f0a02c90700 (LWP 966793))]
(gdb) backtrace
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f0a31acf859 in __GI_abort () at abort.c:79
#2 0x00007f0a31b3a26e in __libc_message (action=action@entry=do_abort, fmt=0x7f0a02c8f230 "\377\377\377\377", fmt@entry=0x7f0a31c64298 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#3 0x00007f0a31b422fc in malloc_printerr (str=str@entry=0x7f0a31c6244d "corrupted size vs. prev_size") at malloc.c:5347
#4 0x00007f0a31b4296b in unlink_chunk (p=p@entry=0x7f09fc03e9d0, av=0x7f09fc000020) at malloc.c:1454
#5 0x00007f0a31b43e8b in _int_free (av=0x7f09fc000020, p=0x7f09fc02e9c0, have_lock=<optimized out>) at malloc.c:4342
#6 0x00007f0a2f241376 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#7 0x00007f0a2f241b3e in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#8 0x00007f0a2f243091 in curl_multi_perform () from /lib/x86_64-linux-gnu/libcurl.so.4
#9 0x00007f0a2f23950b in curl_easy_perform () from /lib/x86_64-linux-gnu/libcurl.so.4
#10 0x00007f0a2b1b46d8 in _dd_curl_send_stack (metrics=0x7f0a02c8f890, stack=<optimized out>, writer=0x7f0a2c047cc0) at /home/circleci/datadog/tmp/build_extension/ext/coms.c:856
#11 _dd_writer_loop (_=<optimized out>) at /home/circleci/datadog/tmp/build_extension/ext/coms.c:1136
#12 0x00007f0a31a90609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x00007f0a31bcc353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
(gdb) thread apply all bt
Thread 2 (Thread 0x7f0a2fbb4980 (LWP 966777)):
#0 futex_abstimed_wait_cancelable (private=<optimized out>, abstime=0x7fff12c4d910, clockid=<optimized out>, expected=0, futex_word=0x55d921798190) at ../sysdeps/nptl/futex-internal.h:320
#1 __pthread_cond_wait_common (abstime=0x7fff12c4d910, clockid=<optimized out>, mutex=0x55d921798140, cond=0x55d921798168) at pthread_cond_wait.c:520
#2 __pthread_cond_timedwait (cond=0x55d921798168, mutex=0x55d921798140, abstime=0x7fff12c4d910) at pthread_cond_wait.c:665
#3 0x00007f0a2b1b5481 in ddtrace_coms_flush_shutdown_writer_synchronous () at /home/circleci/datadog/tmp/build_extension/ext/coms.c:1336
#4 0x00007f0a2b1aa9fa in zm_shutdown_ddtrace (type=<optimized out>, module_number=62) at /home/circleci/datadog/tmp/build_extension/ext/ddtrace.c:1493
#5 0x000055d9204c0fef in module_destructor (module=module@entry=0x55d9214f07a0) at ./Zend/zend_API.c:2563
#6 0x000055d9204bac20 in module_destructor_zval (zv=<optimized out>) at ./Zend/zend.c:761
#7 0x000055d9204cc809 in _zend_hash_del_el_ex (prev=<optimized out>, p=<optimized out>, idx=61, ht=<optimized out>) at ./Zend/zend_hash.c:1305
#8 _zend_hash_del_el (p=0x55d9214d6710, idx=61, ht=0x55d9206be600 <module_registry>) at ./Zend/zend_hash.c:1328
#9 zend_hash_graceful_reverse_destroy (ht=ht@entry=0x55d9206be600 <module_registry>) at ./Zend/zend_hash.c:1782
#10 0x000055d9204bf880 in zend_destroy_modules () at ./Zend/zend_API.c:1995
#11 0x000055d9204bbbb7 in zend_shutdown () at ./Zend/zend.c:1048
#12 0x000055d92045b812 in php_module_shutdown () at ./main/main.c:2478
#13 php_module_shutdown () at ./main/main.c:2455
#14 0x000055d920321b19 in main (argc=4, argv=0x55d92140b2d0) at ./sapi/cli/php_cli.c:1371
Thread 1 (Thread 0x7f0a02c90700 (LWP 966793)):
#0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1 0x00007f0a31acf859 in __GI_abort () at abort.c:79
#2 0x00007f0a31b3a26e in __libc_message (action=action@entry=do_abort, fmt=0x7f0a02c8f230 "\377\377\377\377", fmt@entry=0x7f0a31c64298 "%s\n") at ../sysdeps/posix/libc_fatal.c:156
#3 0x00007f0a31b422fc in malloc_printerr (str=str@entry=0x7f0a31c6244d "corrupted size vs. prev_size") at malloc.c:5347
#4 0x00007f0a31b4296b in unlink_chunk (p=p@entry=0x7f09fc03e9d0, av=0x7f09fc000020) at malloc.c:1454
#5 0x00007f0a31b43e8b in _int_free (av=0x7f09fc000020, p=0x7f09fc02e9c0, have_lock=<optimized out>) at malloc.c:4342
#6 0x00007f0a2f241376 in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#7 0x00007f0a2f241b3e in ?? () from /lib/x86_64-linux-gnu/libcurl.so.4
#8 0x00007f0a2f243091 in curl_multi_perform () from /lib/x86_64-linux-gnu/libcurl.so.4
#9 0x00007f0a2f23950b in curl_easy_perform () from /lib/x86_64-linux-gnu/libcurl.so.4
#10 0x00007f0a2b1b46d8 in _dd_curl_send_stack (metrics=0x7f0a02c8f890, stack=<optimized out>, writer=0x7f0a2c047cc0) at /home/circleci/datadog/tmp/build_extension/ext/coms.c:856
#11 _dd_writer_loop (_=<optimized out>) at /home/circleci/datadog/tmp/build_extension/ext/coms.c:1136
#12 0x00007f0a31a90609 in start_thread (arg=<optimized out>) at pthread_create.c:477
#13 0x00007f0a31bcc353 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
PHP version
PHP 7.4.3-4ubuntu2.24 (cli) (built: Sep 30 2024 18:16:20) ( NTS )
Tracer or profiler version
1.6.4
Installed extensions
$ php -m
[PHP Modules]
apc
apcu
calendar
Core
ctype
curl
date
ddappsec
ddtrace
dom
exif
FFI
fileinfo
filter
ftp
gd
gettext
gree_gadgetproxy_renderer
gree_richtext_renderer
gree_swfdec
hash
http
iconv
intl
json
libmemcached
libxml
mbstring
memcache
msgpack
mysqli
mysqlnd
openssl
pcntl
pcre
PDO
pdo_mysql
Phar
posix
propro
raphf
readline
redis
Reflection
session
shmop
SimpleXML
soap
sockets
sodium
SPL
standard
swfed
sysvmsg
sysvsem
sysvshm
tokenizer
uuid
xml
xmlreader
xmlwriter
xsl
Zend OPcache
zlib
[Zend Modules]
Zend OPcache
ddappsec
ddtrace
Output of phpinfo()
$ php --ri=ddtrace
ddtrace
Datadog PHP tracer extension
For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/
(c) Datadog 2020
Datadog tracing support => enabled
Version => 1.6.4
DATADOG TRACER CONFIGURATION => {
"date": "2025-04-13T17:51:09Z",
"os_name": "Linux ip-10-104-177-140 5.15.0-1071-aws #77~20.04.1-Ubuntu SMP Thu Oct 3 19:39:59 UTC 2024 x86_64",
"os_version": "5.15.0-1071-aws",
"version": "1.6.4",
"lang": "php",
"lang_version": "7.4.33",
"env": null,
"enabled": true,
"service": null,
"enabled_cli": true,
"agent_url": "unix:\/\/\/var\/run\/datadog\/apm.socket",
"debug": false,
"analytics_enabled": false,
"sample_rate": -1,
"sampling_rules": [],
"tags": [],
"service_mapping": [],
"distributed_tracing_enabled": true,
"dd_version": null,
"architecture": "x86_64",
"instrumentation_telemetry_enabled": true,
"sapi": "cli",
"datadog.trace.sources_path": "\/opt\/datadog\/dd-library\/1.6.4\/dd-trace-sources\/src",
"open_basedir_configured": false,
"uri_fragment_regex": null,
"uri_mapping_incoming": null,
"uri_mapping_outgoing": null,
"auto_flush_enabled": true,
"generate_root_span": true,
"http_client_split_by_domain": false,
"measure_compile_time": true,
"report_hostname_on_root_span": false,
"traced_internal_functions": null,
"enabled_from_env": true,
"opcache.file_cache": null,
"sidecar_trace_sender": false
}
Diagnostics
Diagnostic checks => passed
Directive => Local Value => Master Value
ddtrace.disable => 0 => 0
ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup
datadog.trace.sidecar_trace_sender => Off => Off
datadog.trace.sources_path => /opt/datadog/dd-library/1.6.4/dd-trace-sources/src => /opt/datadog/dd-library/1.6.4/dd-trace-sources/src
datadog.autoload_no_compile => Off => Off
datadog.trace.agent_url => no value => no value
datadog.agent_host => no value => no value
datadog.dogstatsd_url => no value => no value
datadog.api_key => no value => no value
datadog.distributed_tracing => On => On
datadog.dogstatsd_port => 8125 => 8125
datadog.env => no value => no value
datadog.autofinish_spans => Off => Off
datadog.trace.url_as_resource_names_enabled => On => On
datadog.http_server_route_based_naming => On => On
datadog.service => no value => no value
datadog.service_mapping => no value => no value
datadog.tags => no value => no value
datadog.trace.agent_port => 0 => 0
datadog.trace.analytics_enabled => Off => Off
datadog.trace.append_trace_ids_to_logs => Off => Off
datadog.trace.auto_flush_enabled => On => On
datadog.trace.cli_enabled => On => On
datadog.trace.measure_compile_time => On => On
datadog.trace.measure_peak_memory_usage => On => On
datadog.trace.debug => Off => Off
datadog.trace.enabled => On => On
datadog.instrumentation_telemetry_enabled => On => On
datadog.trace.health_metrics_enabled => Off => Off
datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001
datadog.trace.db_client_split_by_instance => Off => Off
datadog.trace.http_client_split_by_domain => Off => Off
datadog.trace.redis_client_split_by_host => Off => Off
datadog.exception_replay_enabled => Off => Off
datadog.exception_replay_capture_max_frames => -1 => -1
datadog.exception_replay_capture_interval_seconds => 3600 => 3600
datadog.trace.memory_limit => no value => no value
datadog.trace.report_hostname => Off => Off
datadog.trace.flush_collect_cycles => Off => Off
datadog.trace.kafka_distributed_tracing => On => On
datadog.trace.laravel_queue_distributed_tracing => On => On
datadog.trace.symfony_messenger_distributed_tracing => On => On
datadog.trace.symfony_messenger_middlewares => Off => Off
datadog.trace.remove_root_span_laravel_queue => On => On
datadog.trace.remove_root_span_symfony_messenger => On => On
datadog.appsec.rasp_enabled => Off => Off
datadog.trace.remove_autoinstrumentation_orphans => Off => Off
datadog.trace.resource_uri_fragment_regex => no value => no value
datadog.trace.resource_uri_mapping_incoming => no value => no value
datadog.trace.resource_uri_mapping_outgoing => no value => no value
datadog.trace.resource_uri_query_param_allowed => no value => no value
datadog.trace.http_url_query_param_allowed => * => *
datadog.trace.http_post_data_param_allowed => no value => no value
datadog.trace.rate_limit => 100 => 100
datadog.trace.sample_rate => -1 => -1
datadog.trace.sampling_rules => [] => []
datadog.trace.sampling_rules_format => glob => glob
datadog.span_sampling_rules => [] => []
datadog.span_sampling_rules_file => no value => no value
datadog.trace.header_tags => no value => no value
datadog.trace.x_datadog_tags_max_length => 512 => 512
datadog.trace.peer_service_mapping => no value => no value
datadog.trace.peer_service_defaults_enabled => Off => Off
datadog.trace.remove_integration_service_names_enabled => Off => Off
datadog.trace.propagate_service => Off => Off
datadog.trace.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header
datadog.trace.propagation_style_inject => datadog,tracecontext => datadog,tracecontext
datadog.trace.propagation_style => datadog,tracecontext => datadog,tracecontext
datadog.trace.traced_internal_functions => no value => no value
datadog.trace.agent_timeout => 500 => 500
datadog.trace.agent_connect_timeout => 100 => 100
datadog.trace.debug_prng_seed => -1 => -1
datadog.log_backtrace => Off => Off
datadog.crashtracking_enabled => On => On
datadog.trace.generate_root_span => On => On
datadog.trace.spans_limit => 1000 => 1000
datadog.trace.128_bit_traceid_generation_enabled => On => On
datadog.trace.128_bit_traceid_logging_enabled => Off => Off
datadog.trace.bgs_connect_timeout => 2000 => 2000
datadog.trace.bgs_timeout => 5000 => 5000
datadog.trace.agent_flush_interval => 5000 => 5000
datadog.telemetry_heartbeat_interval => 60 => 60
datadog.trace.agent_flush_after_n_requests => 10 => 10
datadog.trace.shutdown_timeout => 5000 => 5000
datadog.trace.startup_logs => On => On
datadog.trace.once_logs => On => On
datadog.trace.agent_retries => 0 => 0
datadog.trace.agent_debug_verbose_curl => Off => Off
datadog.trace.debug_curl_output => Off => Off
datadog.trace.beta_high_memory_pressure_percent => 80 => 80
datadog.trace.agentless => Off => Off
datadog.trace.warn_legacy_dd_trace => On => On
datadog.trace.retain_thread_capabilities => Off => Off
datadog.version => no value => no value
datadog.trace.obfuscation_query_string_regexp => (?i)(?:(?:"|%22)?)(?:(?:old[-_]?|new[-_]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-_]?phrase)?|secret|(?:api[-_]?|private[-_]?|public[-_]?|access[-_]?|secret[-_]?|app(?:lication)?[-_]?)key(?:[-_]?id)?|token|consumer[-_]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9._\-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^\-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9._-]+)?) => (?i)(?:(?:"|%22)?)(?:(?:old[-_]?|new[-_]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-_]?phrase)?|secret|(?:api[-_]?|private[-_]?|public[-_]?|access[-_]?|secret[-_]?|app(?:lication)?[-_]?)key(?:[-_]?id)?|token|consumer[-_]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)*(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)*(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9._\-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu]_[0-9a-zA-Z]{36}|ey[I-L](?:[\w=-]|%3D)+\.ey[I-L](?:[\w=-]|%3D)+(?:\.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^\-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9._-]+)?)
datadog.trace.memcached_obfuscation => On => On
datadog.trace.client_ip_enabled => Off => Off
datadog.trace.client_ip_header => no value => no value
datadog.trace.forked_process => On => On
datadog.trace.hook_limit => 100 => 100
datadog.trace.buffer_size => 2097152 => 2097152
datadog.trace.agent_max_payload_size => 52428800 => 52428800
datadog.trace.agent_stack_initial_size => 131072 => 131072
datadog.trace.agent_stack_backlog => 12 => 12
datadog.trace.agent_test_session_token => no value => no value
datadog.trace.propagate_user_id_default => Off => Off
datadog.dbm_propagation_mode => disabled => disabled
datadog.trace.wordpress_additional_actions => no value => no value
datadog.trace.wordpress_callbacks => On => On
datadog.integration_metrics_enabled => On => On
datadog.trace.otel_enabled => Off => Off
datadog.trace.log_file => no value => no value
datadog.trace.log_level => error => error
datadog.appsec.sca_enabled => Off => Off
datadog.trace.git_metadata_enabled => On => On
datadog.git_commit_sha => no value => no value
datadog.git_repository_url => no value => no value
datadog.openai_service => no value => no value
datadog.openai_metrics_enabled => On => On
datadog.openai_logs_enabled => Off => Off
datadog.openai_span_char_limit => 128 => 128
datadog.openai_span_prompt_completion_sample_rate => 1.0 => 1.0
datadog.openai_log_prompt_completion_sample_rate => 0.1 => 0.1
datadog.inject_force => Off => Off
datadog.remote_config_poll_interval_seconds => 5 => 5
datadog.remote_config_enabled => On => On
datadog.dynamic_instrumentation.enabled => Off => Off
datadog.dynamic_instrumentation.redacted_identifiers => no value => no value
datadog.experimental_appsec_standalone_enabled => Off => Off
datadog.dynamic_instrumentation.redacted_types => no value => no value
datadog.trace.amqp_enabled => On => On
datadog.trace.amqp_analytics_enabled => Off => Off
datadog.amqp_analytics_enabled => Off => Off
datadog.trace.amqp_analytics_sample_rate => 1 => 1
datadog.amqp_analytics_sample_rate => 1 => 1
datadog.trace.cakephp_enabled => On => On
datadog.trace.cakephp_analytics_enabled => Off => Off
datadog.cakephp_analytics_enabled => Off => Off
datadog.trace.cakephp_analytics_sample_rate => 1 => 1
datadog.cakephp_analytics_sample_rate => 1 => 1
datadog.trace.codeigniter_enabled => On => On
datadog.trace.codeigniter_analytics_enabled => Off => Off
datadog.codeigniter_analytics_enabled => Off => Off
datadog.trace.codeigniter_analytics_sample_rate => 1 => 1
datadog.codeigniter_analytics_sample_rate => 1 => 1
datadog.trace.exec_enabled => On => On
datadog.trace.exec_analytics_enabled => Off => Off
datadog.exec_analytics_enabled => Off => Off
datadog.trace.exec_analytics_sample_rate => 1 => 1
datadog.exec_analytics_sample_rate => 1 => 1
datadog.trace.curl_enabled => On => On
datadog.trace.curl_analytics_enabled => Off => Off
datadog.curl_analytics_enabled => Off => Off
datadog.trace.curl_analytics_sample_rate => 1 => 1
datadog.curl_analytics_sample_rate => 1 => 1
datadog.trace.drupal_enabled => On => On
datadog.trace.drupal_analytics_enabled => Off => Off
datadog.drupal_analytics_enabled => Off => Off
datadog.trace.drupal_analytics_sample_rate => 1 => 1
datadog.drupal_analytics_sample_rate => 1 => 1
datadog.trace.elasticsearch_enabled => On => On
datadog.trace.elasticsearch_analytics_enabled => Off => Off
datadog.elasticsearch_analytics_enabled => Off => Off
datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1
datadog.elasticsearch_analytics_sample_rate => 1 => 1
datadog.trace.eloquent_enabled => On => On
datadog.trace.eloquent_analytics_enabled => Off => Off
datadog.eloquent_analytics_enabled => Off => Off
datadog.trace.eloquent_analytics_sample_rate => 1 => 1
datadog.eloquent_analytics_sample_rate => 1 => 1
datadog.trace.filesystem_enabled => On => On
datadog.trace.filesystem_analytics_enabled => Off => Off
datadog.filesystem_analytics_enabled => Off => Off
datadog.trace.filesystem_analytics_sample_rate => 1 => 1
datadog.filesystem_analytics_sample_rate => 1 => 1
datadog.trace.frankenphp_enabled => On => On
datadog.trace.frankenphp_analytics_enabled => Off => Off
datadog.frankenphp_analytics_enabled => Off => Off
datadog.trace.frankenphp_analytics_sample_rate => 1 => 1
datadog.frankenphp_analytics_sample_rate => 1 => 1
datadog.trace.googlespanner_enabled => On => On
datadog.trace.googlespanner_analytics_enabled => Off => Off
datadog.googlespanner_analytics_enabled => Off => Off
datadog.trace.googlespanner_analytics_sample_rate => 1 => 1
datadog.googlespanner_analytics_sample_rate => 1 => 1
datadog.trace.guzzle_enabled => On => On
datadog.trace.guzzle_analytics_enabled => Off => Off
datadog.guzzle_analytics_enabled => Off => Off
datadog.trace.guzzle_analytics_sample_rate => 1 => 1
datadog.guzzle_analytics_sample_rate => 1 => 1
datadog.trace.kafka_enabled => On => On
datadog.trace.kafka_analytics_enabled => Off => Off
datadog.kafka_analytics_enabled => Off => Off
datadog.trace.kafka_analytics_sample_rate => 1 => 1
datadog.kafka_analytics_sample_rate => 1 => 1
datadog.trace.laminas_enabled => On => On
datadog.trace.laminas_analytics_enabled => Off => Off
datadog.laminas_analytics_enabled => Off => Off
datadog.trace.laminas_analytics_sample_rate => 1 => 1
datadog.laminas_analytics_sample_rate => 1 => 1
datadog.trace.laravel_enabled => On => On
datadog.trace.laravel_analytics_enabled => Off => Off
datadog.laravel_analytics_enabled => Off => Off
datadog.trace.laravel_analytics_sample_rate => 1 => 1
datadog.laravel_analytics_sample_rate => 1 => 1
datadog.trace.laravelqueue_enabled => On => On
datadog.trace.laravelqueue_analytics_enabled => Off => Off
datadog.laravelqueue_analytics_enabled => Off => Off
datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1
datadog.laravelqueue_analytics_sample_rate => 1 => 1
datadog.trace.logs_enabled => Off => Off
datadog.logs_injection => Off => Off
datadog.trace.logs_analytics_enabled => Off => Off
datadog.logs_analytics_enabled => Off => Off
datadog.trace.logs_analytics_sample_rate => 1 => 1
datadog.logs_analytics_sample_rate => 1 => 1
datadog.trace.lumen_enabled => On => On
datadog.trace.lumen_analytics_enabled => Off => Off
datadog.lumen_analytics_enabled => Off => Off
datadog.trace.lumen_analytics_sample_rate => 1 => 1
datadog.lumen_analytics_sample_rate => 1 => 1
datadog.trace.magento_enabled => On => On
datadog.trace.magento_analytics_enabled => Off => Off
datadog.magento_analytics_enabled => Off => Off
datadog.trace.magento_analytics_sample_rate => 1 => 1
datadog.magento_analytics_sample_rate => 1 => 1
datadog.trace.memcache_enabled => On => On
datadog.trace.memcache_analytics_enabled => Off => Off
datadog.memcache_analytics_enabled => Off => Off
datadog.trace.memcache_analytics_sample_rate => 1 => 1
datadog.memcache_analytics_sample_rate => 1 => 1
datadog.trace.memcached_enabled => On => On
datadog.trace.memcached_analytics_enabled => Off => Off
datadog.memcached_analytics_enabled => Off => Off
datadog.trace.memcached_analytics_sample_rate => 1 => 1
datadog.memcached_analytics_sample_rate => 1 => 1
datadog.trace.mongo_enabled => On => On
datadog.trace.mongo_analytics_enabled => Off => Off
datadog.mongo_analytics_enabled => Off => Off
datadog.trace.mongo_analytics_sample_rate => 1 => 1
datadog.mongo_analytics_sample_rate => 1 => 1
datadog.trace.mongodb_enabled => On => On
datadog.trace.mongodb_analytics_enabled => Off => Off
datadog.mongodb_analytics_enabled => Off => Off
datadog.trace.mongodb_analytics_sample_rate => 1 => 1
datadog.mongodb_analytics_sample_rate => 1 => 1
datadog.trace.mysqli_enabled => On => On
datadog.trace.mysqli_analytics_enabled => Off => Off
datadog.mysqli_analytics_enabled => Off => Off
datadog.trace.mysqli_analytics_sample_rate => 1 => 1
datadog.mysqli_analytics_sample_rate => 1 => 1
datadog.trace.nette_enabled => On => On
datadog.trace.nette_analytics_enabled => Off => Off
datadog.nette_analytics_enabled => Off => Off
datadog.trace.nette_analytics_sample_rate => 1 => 1
datadog.nette_analytics_sample_rate => 1 => 1
datadog.trace.openai_enabled => On => On
datadog.trace.openai_analytics_enabled => Off => Off
datadog.openai_analytics_enabled => Off => Off
datadog.trace.openai_analytics_sample_rate => 1 => 1
datadog.openai_analytics_sample_rate => 1 => 1
datadog.trace.pcntl_enabled => On => On
datadog.trace.pcntl_analytics_enabled => Off => Off
datadog.pcntl_analytics_enabled => Off => Off
datadog.trace.pcntl_analytics_sample_rate => 1 => 1
datadog.pcntl_analytics_sample_rate => 1 => 1
datadog.trace.pdo_enabled => On => On
datadog.trace.pdo_analytics_enabled => Off => Off
datadog.pdo_analytics_enabled => Off => Off
datadog.trace.pdo_analytics_sample_rate => 1 => 1
datadog.pdo_analytics_sample_rate => 1 => 1
datadog.trace.phpredis_enabled => On => On
datadog.trace.phpredis_analytics_enabled => Off => Off
datadog.phpredis_analytics_enabled => Off => Off
datadog.trace.phpredis_analytics_sample_rate => 1 => 1
datadog.phpredis_analytics_sample_rate => 1 => 1
datadog.trace.predis_enabled => On => On
datadog.trace.predis_analytics_enabled => Off => Off
datadog.predis_analytics_enabled => Off => Off
datadog.trace.predis_analytics_sample_rate => 1 => 1
datadog.predis_analytics_sample_rate => 1 => 1
datadog.trace.psr18_enabled => On => On
datadog.trace.psr18_analytics_enabled => Off => Off
datadog.psr18_analytics_enabled => Off => Off
datadog.trace.psr18_analytics_sample_rate => 1 => 1
datadog.psr18_analytics_sample_rate => 1 => 1
datadog.trace.roadrunner_enabled => On => On
datadog.trace.roadrunner_analytics_enabled => Off => Off
datadog.roadrunner_analytics_enabled => Off => Off
datadog.trace.roadrunner_analytics_sample_rate => 1 => 1
datadog.roadrunner_analytics_sample_rate => 1 => 1
datadog.trace.sqlsrv_enabled => On => On
datadog.trace.sqlsrv_analytics_enabled => Off => Off
datadog.sqlsrv_analytics_enabled => Off => Off
datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1
datadog.sqlsrv_analytics_sample_rate => 1 => 1
datadog.trace.slim_enabled => On => On
datadog.trace.slim_analytics_enabled => Off => Off
datadog.slim_analytics_enabled => Off => Off
datadog.trace.slim_analytics_sample_rate => 1 => 1
datadog.slim_analytics_sample_rate => 1 => 1
datadog.trace.swoole_enabled => On => On
datadog.trace.swoole_analytics_enabled => Off => Off
datadog.swoole_analytics_enabled => Off => Off
datadog.trace.swoole_analytics_sample_rate => 1 => 1
datadog.swoole_analytics_sample_rate => 1 => 1
datadog.trace.symfony_enabled => On => On
datadog.trace.symfony_analytics_enabled => Off => Off
datadog.symfony_analytics_enabled => Off => Off
datadog.trace.symfony_analytics_sample_rate => 1 => 1
datadog.symfony_analytics_sample_rate => 1 => 1
datadog.trace.symfonymessenger_enabled => On => On
datadog.trace.symfonymessenger_analytics_enabled => Off => Off
datadog.symfonymessenger_analytics_enabled => Off => Off
datadog.trace.symfonymessenger_analytics_sample_rate => 1 => 1
datadog.symfonymessenger_analytics_sample_rate => 1 => 1
datadog.trace.web_enabled => On => On
datadog.trace.web_analytics_enabled => Off => Off
datadog.web_analytics_enabled => Off => Off
datadog.trace.web_analytics_sample_rate => 1 => 1
datadog.web_analytics_sample_rate => 1 => 1
datadog.trace.wordpress_enabled => On => On
datadog.trace.wordpress_analytics_enabled => Off => Off
datadog.wordpress_analytics_enabled => Off => Off
datadog.trace.wordpress_analytics_sample_rate => 1 => 1
datadog.wordpress_analytics_sample_rate => 1 => 1
datadog.trace.yii_enabled => On => On
datadog.trace.yii_analytics_enabled => Off => Off
datadog.yii_analytics_enabled => Off => Off
datadog.trace.yii_analytics_sample_rate => 1 => 1
datadog.yii_analytics_sample_rate => 1 => 1
datadog.trace.zendframework_enabled => On => On
datadog.trace.zendframework_analytics_enabled => Off => Off
datadog.zendframework_analytics_enabled => Off => Off
datadog.trace.zendframework_analytics_sample_rate => 1 => 1
datadog.zendframework_analytics_sample_rate => 1 => 1
Upgrading from
No response
Hey @junjihashimoto,
thanks for the bug report, the main question here is: why is _dd_writer_loop started a second time, while there's already an active one?
I haven't seen this happening and am not sure why/what would cause this.
@bwoebi Hi,
I don't think there are two writers; I think the main thread just starts the writer thread and waits for it to finish using join.
@junjihashimoto Is it possible for you to reproduce this with valgrind or asan? Thanks!
@bwoebi The error occurs in production, so I can not use it. Based on the situation, I think it's a memory management issue with stack, because the only things allocating and freeing memory in that code are stack and curl.
It is possible to log using ebpf, but there is no urprobe for that memory in ddtrace.so.