dd-trace-php [Bug]: Spamming LOG: [ddtrace] [error] Failed sending traces to the sidecar: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

Bug report

Hello,

I am currently using this version

PHP 8.4.2
Datadog agent version: 7.60.1
PHP Tracer library 1.6.2

After some time which can vary I get spammed in my logs. I receive thousands of error messages: [ddtrace] [error] Failed sending traces to the sidecar: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

I forced a new deployment of my application and it stopped right now but after some time I just get the same issue again.

Can anyone help ?

PHP version

8.4.2

Tracer or profiler version

1.6.2

Installed extensions

[PHP Modules] amqp bcmath calendar Core ctype curl date ddappsec ddtrace dom exif FFI fileinfo filter ftp gettext hash iconv json libxml mbstring mysqli mysqlnd openssl pcntl pcre PDO pdo_mysql Phar posix random readline Reflection session shmop SimpleXML soap sockets sodium SPL standard sysvmsg sysvsem sysvshm tokenizer xml xmlreader xmlwriter xsl Zend OPcache zlib

[Zend Modules] Zend OPcache ddappsec ddtrace

Output of `phpinfo()`

ddtrace

Datadog PHP tracer extension For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/ (c) Datadog 2020

Datadog tracing support => enabled Version => 1.6.2 DATADOG TRACER CONFIGURATION => { "date": "2025-01-15T10:53:14Z", "os_name": "", "os_version": "", "version": "1.6.2", "lang": "php", "lang_version": "8.4.1", "env": "production", "enabled": true, "service": "", "enabled_cli": true, "agent_url": "http://localhost:8126", "debug": false, "analytics_enabled": false, "sample_rate": -1, "sampling_rules": [], "tags": { "service": "" }, "service_mapping": { "pdo": "", "redis": "" }, "distributed_tracing_enabled": true, "dd_version": "", "architecture": "x86_64", "instrumentation_telemetry_enabled": true, "sapi": "cli", "datadog.trace.sources_path": "/opt/datadog/dd-library/1.6.2/dd-trace-sources/src", "open_basedir_configured": false, "uri_fragment_regex": null, "uri_mapping_incoming": null, "uri_mapping_outgoing": null, "auto_flush_enabled": true, "generate_root_span": true, "http_client_split_by_domain": false, "measure_compile_time": true, "report_hostname_on_root_span": false, "traced_internal_functions": null, "enabled_from_env": true, "opcache.file_cache": null, "sidecar_trace_sender": true }

                           Diagnostics

Diagnostic checks => passed

Directive => Local Value => Master Value ddtrace.disable => 0 => 0 ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup datadog.trace.sidecar_trace_sender => On => On datadog.trace.sources_path => /opt/datadog/dd-library/1.6.2/dd-trace-sources/src => /opt/datadog/dd-library/1.6.2/dd-trace-sources/src datadog.autoload_no_compile => Off => Off datadog.trace.agent_url => no value => no value datadog.agent_host => no value => no value datadog.dogstatsd_url => no value => no value datadog.api_key => no value => no value datadog.distributed_tracing => On => On datadog.dogstatsd_port => 8125 => 8125 datadog.env => production => production datadog.autofinish_spans => Off => Off datadog.trace.url_as_resource_names_enabled => On => On datadog.http_server_route_based_naming => On => On datadog.service => datadog.service_mapping => datadog.tags => datadog.trace.agent_port => 0 => 0 datadog.trace.analytics_enabled => Off => Off datadog.trace.append_trace_ids_to_logs => Off => Off datadog.trace.auto_flush_enabled => On => On datadog.trace.cli_enabled => On => On datadog.trace.measure_compile_time => On => On datadog.trace.measure_peak_memory_usage => On => On datadog.trace.debug => Off => Off datadog.trace.enabled => On => On datadog.instrumentation_telemetry_enabled => On => On datadog.trace.health_metrics_enabled => Off => Off datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001 datadog.trace.db_client_split_by_instance => Off => Off datadog.trace.http_client_split_by_domain => Off => Off datadog.trace.redis_client_split_by_host => Off => Off datadog.exception_replay_enabled => Off => Off datadog.exception_replay_capture_max_frames => -1 => -1 datadog.exception_replay_capture_interval_seconds => 3600 => 3600 datadog.trace.memory_limit => no value => no value datadog.trace.report_hostname => Off => Off datadog.trace.flush_collect_cycles => Off => Off datadog.trace.kafka_distributed_tracing => On => On datadog.trace.laravel_queue_distributed_tracing => On => On datadog.trace.symfony_messenger_distributed_tracing => On => On datadog.trace.symfony_messenger_middlewares => Off => Off datadog.trace.remove_root_span_laravel_queue => On => On datadog.trace.remove_root_span_symfony_messenger => On => On datadog.appsec.rasp_enabled => Off => Off datadog.trace.remove_autoinstrumentation_orphans => Off => Off datadog.trace.resource_uri_fragment_regex => no value => no value datadog.trace.resource_uri_mapping_incoming => no value => no value datadog.trace.resource_uri_mapping_outgoing => no value => no value datadog.trace.resource_uri_query_param_allowed => no value => no value datadog.trace.http_url_query_param_allowed => * => * datadog.trace.http_post_data_param_allowed => no value => no value datadog.trace.rate_limit => 100 => 100 datadog.trace.sample_rate => -1 => -1 datadog.trace.sampling_rules => [] => [] datadog.trace.sampling_rules_format => glob => glob datadog.span_sampling_rules => [] => [] datadog.span_sampling_rules_file => no value => no value datadog.trace.header_tags => no value => no value datadog.trace.x_datadog_tags_max_length => 512 => 512 datadog.trace.peer_service_mapping => no value => no value datadog.trace.peer_service_defaults_enabled => Off => Off datadog.trace.remove_integration_service_names_enabled => Off => Off datadog.trace.propagate_service => Off => Off datadog.trace.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header datadog.trace.propagation_style_inject => datadog,tracecontext => datadog,tracecontext datadog.trace.propagation_style => datadog,tracecontext => datadog,tracecontext datadog.trace.traced_internal_functions => no value => no value datadog.trace.agent_timeout => 500 => 500 datadog.trace.agent_connect_timeout => 100 => 100 datadog.trace.debug_prng_seed => -1 => -1 datadog.log_backtrace => Off => Off datadog.crashtracking_enabled => On => On datadog.trace.generate_root_span => On => On datadog.trace.spans_limit => 1000 => 1000 datadog.trace.128_bit_traceid_generation_enabled => On => On datadog.trace.128_bit_traceid_logging_enabled => Off => Off datadog.trace.bgs_connect_timeout => 2000 => 2000 datadog.trace.bgs_timeout => 5000 => 5000 datadog.trace.agent_flush_interval => 1001 => 1001 datadog.telemetry_heartbeat_interval => 60 => 60 datadog.trace.agent_flush_after_n_requests => 0 => 0 datadog.trace.shutdown_timeout => 5000 => 5000 datadog.trace.startup_logs => On => On datadog.trace.once_logs => On => On datadog.trace.agent_retries => 0 => 0 datadog.trace.agent_debug_verbose_curl => Off => Off datadog.trace.debug_curl_output => Off => Off datadog.trace.beta_high_memory_pressure_percent => 80 => 80 datadog.trace.agentless => Off => Off datadog.trace.warn_legacy_dd_trace => On => On datadog.trace.retain_thread_capabilities => Off => Off datadog.version => datadog.trace.obfuscation_query_string_regexp => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9.-]+)?) => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9.-]+)?) datadog.trace.memcached_obfuscation => On => On datadog.trace.client_ip_enabled => Off => Off datadog.trace.client_ip_header => no value => no value datadog.trace.forked_process => On => On datadog.trace.hook_limit => 100 => 100 datadog.trace.buffer_size => 2097152 => 2097152 datadog.trace.agent_max_payload_size => 52428800 => 52428800 datadog.trace.agent_stack_initial_size => 131072 => 131072 datadog.trace.agent_stack_backlog => 12 => 12 datadog.trace.agent_test_session_token => no value => no value datadog.trace.propagate_user_id_default => Off => Off datadog.dbm_propagation_mode => disabled => disabled datadog.trace.wordpress_additional_actions => no value => no value datadog.trace.wordpress_callbacks => On => On datadog.integration_metrics_enabled => On => On datadog.trace.otel_enabled => Off => Off datadog.trace.log_file => no value => no value datadog.trace.log_level => error => error datadog.appsec.sca_enabled => Off => Off datadog.trace.git_metadata_enabled => On => On datadog.git_commit_sha => no value => no value datadog.git_repository_url => no value => no value datadog.openai_service => no value => no value datadog.openai_metrics_enabled => On => On datadog.openai_logs_enabled => Off => Off datadog.openai_span_char_limit => 128 => 128 datadog.openai_span_prompt_completion_sample_rate => 1.0 => 1.0 datadog.openai_log_prompt_completion_sample_rate => 0.1 => 0.1 datadog.inject_force => Off => Off datadog.remote_config_poll_interval_seconds => 5 => 5 datadog.remote_config_enabled => Off => Off datadog.dynamic_instrumentation.enabled => Off => Off datadog.dynamic_instrumentation.redacted_identifiers => no value => no value datadog.experimental_appsec_standalone_enabled => Off => Off datadog.dynamic_instrumentation.redacted_types => no value => no value datadog.trace.amqp_enabled => On => On datadog.trace.amqp_analytics_enabled => Off => Off datadog.amqp_analytics_enabled => Off => Off datadog.trace.amqp_analytics_sample_rate => 1 => 1 datadog.amqp_analytics_sample_rate => 1 => 1 datadog.trace.cakephp_enabled => On => On datadog.trace.cakephp_analytics_enabled => Off => Off datadog.cakephp_analytics_enabled => Off => Off datadog.trace.cakephp_analytics_sample_rate => 1 => 1 datadog.cakephp_analytics_sample_rate => 1 => 1 datadog.trace.codeigniter_enabled => On => On datadog.trace.codeigniter_analytics_enabled => Off => Off datadog.codeigniter_analytics_enabled => Off => Off datadog.trace.codeigniter_analytics_sample_rate => 1 => 1 datadog.codeigniter_analytics_sample_rate => 1 => 1 datadog.trace.exec_enabled => On => On datadog.trace.exec_analytics_enabled => Off => Off datadog.exec_analytics_enabled => Off => Off datadog.trace.exec_analytics_sample_rate => 1 => 1 datadog.exec_analytics_sample_rate => 1 => 1 datadog.trace.curl_enabled => On => On datadog.trace.curl_analytics_enabled => Off => Off datadog.curl_analytics_enabled => Off => Off datadog.trace.curl_analytics_sample_rate => 1 => 1 datadog.curl_analytics_sample_rate => 1 => 1 datadog.trace.drupal_enabled => On => On datadog.trace.drupal_analytics_enabled => Off => Off datadog.drupal_analytics_enabled => Off => Off datadog.trace.drupal_analytics_sample_rate => 1 => 1 datadog.drupal_analytics_sample_rate => 1 => 1 datadog.trace.elasticsearch_enabled => On => On datadog.trace.elasticsearch_analytics_enabled => Off => Off datadog.elasticsearch_analytics_enabled => Off => Off datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1 datadog.elasticsearch_analytics_sample_rate => 1 => 1 datadog.trace.eloquent_enabled => On => On datadog.trace.eloquent_analytics_enabled => Off => Off datadog.eloquent_analytics_enabled => Off => Off datadog.trace.eloquent_analytics_sample_rate => 1 => 1 datadog.eloquent_analytics_sample_rate => 1 => 1 datadog.trace.filesystem_enabled => On => On datadog.trace.filesystem_analytics_enabled => Off => Off datadog.filesystem_analytics_enabled => Off => Off datadog.trace.filesystem_analytics_sample_rate => 1 => 1 datadog.filesystem_analytics_sample_rate => 1 => 1 datadog.trace.frankenphp_enabled => On => On datadog.trace.frankenphp_analytics_enabled => Off => Off datadog.frankenphp_analytics_enabled => Off => Off datadog.trace.frankenphp_analytics_sample_rate => 1 => 1 datadog.frankenphp_analytics_sample_rate => 1 => 1 datadog.trace.googlespanner_enabled => On => On datadog.trace.googlespanner_analytics_enabled => Off => Off datadog.googlespanner_analytics_enabled => Off => Off datadog.trace.googlespanner_analytics_sample_rate => 1 => 1 datadog.googlespanner_analytics_sample_rate => 1 => 1 datadog.trace.guzzle_enabled => On => On datadog.trace.guzzle_analytics_enabled => Off => Off datadog.guzzle_analytics_enabled => Off => Off datadog.trace.guzzle_analytics_sample_rate => 1 => 1 datadog.guzzle_analytics_sample_rate => 1 => 1 datadog.trace.kafka_enabled => On => On datadog.trace.kafka_analytics_enabled => Off => Off datadog.kafka_analytics_enabled => Off => Off datadog.trace.kafka_analytics_sample_rate => 1 => 1 datadog.kafka_analytics_sample_rate => 1 => 1 datadog.trace.laminas_enabled => On => On datadog.trace.laminas_analytics_enabled => Off => Off datadog.laminas_analytics_enabled => Off => Off datadog.trace.laminas_analytics_sample_rate => 1 => 1 datadog.laminas_analytics_sample_rate => 1 => 1 datadog.trace.laravel_enabled => On => On datadog.trace.laravel_analytics_enabled => Off => Off datadog.laravel_analytics_enabled => Off => Off datadog.trace.laravel_analytics_sample_rate => 1 => 1 datadog.laravel_analytics_sample_rate => 1 => 1 datadog.trace.laravelqueue_enabled => On => On datadog.trace.laravelqueue_analytics_enabled => Off => Off datadog.laravelqueue_analytics_enabled => Off => Off datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1 datadog.laravelqueue_analytics_sample_rate => 1 => 1 datadog.trace.logs_enabled => Off => Off datadog.logs_injection => Off => Off datadog.trace.logs_analytics_enabled => Off => Off datadog.logs_analytics_enabled => Off => Off datadog.trace.logs_analytics_sample_rate => 1 => 1 datadog.logs_analytics_sample_rate => 1 => 1 datadog.trace.lumen_enabled => On => On datadog.trace.lumen_analytics_enabled => Off => Off datadog.lumen_analytics_enabled => Off => Off datadog.trace.lumen_analytics_sample_rate => 1 => 1 datadog.lumen_analytics_sample_rate => 1 => 1 datadog.trace.magento_enabled => On => On datadog.trace.magento_analytics_enabled => Off => Off datadog.magento_analytics_enabled => Off => Off datadog.trace.magento_analytics_sample_rate => 1 => 1 datadog.magento_analytics_sample_rate => 1 => 1 datadog.trace.memcache_enabled => On => On datadog.trace.memcache_analytics_enabled => Off => Off datadog.memcache_analytics_enabled => Off => Off datadog.trace.memcache_analytics_sample_rate => 1 => 1 datadog.memcache_analytics_sample_rate => 1 => 1 datadog.trace.memcached_enabled => On => On datadog.trace.memcached_analytics_enabled => Off => Off datadog.memcached_analytics_enabled => Off => Off datadog.trace.memcached_analytics_sample_rate => 1 => 1 datadog.memcached_analytics_sample_rate => 1 => 1 datadog.trace.mongo_enabled => On => On datadog.trace.mongo_analytics_enabled => Off => Off datadog.mongo_analytics_enabled => Off => Off datadog.trace.mongo_analytics_sample_rate => 1 => 1 datadog.mongo_analytics_sample_rate => 1 => 1 datadog.trace.mongodb_enabled => On => On datadog.trace.mongodb_analytics_enabled => Off => Off datadog.mongodb_analytics_enabled => Off => Off datadog.trace.mongodb_analytics_sample_rate => 1 => 1 datadog.mongodb_analytics_sample_rate => 1 => 1 datadog.trace.mysqli_enabled => On => On datadog.trace.mysqli_analytics_enabled => Off => Off datadog.mysqli_analytics_enabled => Off => Off datadog.trace.mysqli_analytics_sample_rate => 1 => 1 datadog.mysqli_analytics_sample_rate => 1 => 1 datadog.trace.nette_enabled => On => On datadog.trace.nette_analytics_enabled => Off => Off datadog.nette_analytics_enabled => Off => Off datadog.trace.nette_analytics_sample_rate => 1 => 1 datadog.nette_analytics_sample_rate => 1 => 1 datadog.trace.openai_enabled => On => On datadog.trace.openai_analytics_enabled => Off => Off datadog.openai_analytics_enabled => Off => Off datadog.trace.openai_analytics_sample_rate => 1 => 1 datadog.openai_analytics_sample_rate => 1 => 1 datadog.trace.pcntl_enabled => On => On datadog.trace.pcntl_analytics_enabled => Off => Off datadog.pcntl_analytics_enabled => Off => Off datadog.trace.pcntl_analytics_sample_rate => 1 => 1 datadog.pcntl_analytics_sample_rate => 1 => 1 datadog.trace.pdo_enabled => On => On datadog.trace.pdo_analytics_enabled => Off => Off datadog.pdo_analytics_enabled => Off => Off datadog.trace.pdo_analytics_sample_rate => 1 => 1 datadog.pdo_analytics_sample_rate => 1 => 1 datadog.trace.phpredis_enabled => On => On datadog.trace.phpredis_analytics_enabled => Off => Off datadog.phpredis_analytics_enabled => Off => Off datadog.trace.phpredis_analytics_sample_rate => 1 => 1 datadog.phpredis_analytics_sample_rate => 1 => 1 datadog.trace.predis_enabled => On => On datadog.trace.predis_analytics_enabled => Off => Off datadog.predis_analytics_enabled => Off => Off datadog.trace.predis_analytics_sample_rate => 1 => 1 datadog.predis_analytics_sample_rate => 1 => 1 datadog.trace.psr18_enabled => On => On datadog.trace.psr18_analytics_enabled => Off => Off datadog.psr18_analytics_enabled => Off => Off datadog.trace.psr18_analytics_sample_rate => 1 => 1 datadog.psr18_analytics_sample_rate => 1 => 1 datadog.trace.roadrunner_enabled => On => On datadog.trace.roadrunner_analytics_enabled => Off => Off datadog.roadrunner_analytics_enabled => Off => Off datadog.trace.roadrunner_analytics_sample_rate => 1 => 1 datadog.roadrunner_analytics_sample_rate => 1 => 1 datadog.trace.sqlsrv_enabled => On => On datadog.trace.sqlsrv_analytics_enabled => Off => Off datadog.sqlsrv_analytics_enabled => Off => Off datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1 datadog.sqlsrv_analytics_sample_rate => 1 => 1 datadog.trace.slim_enabled => On => On datadog.trace.slim_analytics_enabled => Off => Off datadog.slim_analytics_enabled => Off => Off datadog.trace.slim_analytics_sample_rate => 1 => 1 datadog.slim_analytics_sample_rate => 1 => 1 datadog.trace.swoole_enabled => On => On datadog.trace.swoole_analytics_enabled => Off => Off datadog.swoole_analytics_enabled => Off => Off datadog.trace.swoole_analytics_sample_rate => 1 => 1 datadog.swoole_analytics_sample_rate => 1 => 1 datadog.trace.symfony_enabled => On => On datadog.trace.symfony_analytics_enabled => Off => Off datadog.symfony_analytics_enabled => Off => Off datadog.trace.symfony_analytics_sample_rate => 1 => 1 datadog.symfony_analytics_sample_rate => 1 => 1 datadog.trace.symfonymessenger_enabled => On => On datadog.trace.symfonymessenger_analytics_enabled => Off => Off datadog.symfonymessenger_analytics_enabled => Off => Off datadog.trace.symfonymessenger_analytics_sample_rate => 1 => 1 datadog.symfonymessenger_analytics_sample_rate => 1 => 1 datadog.trace.web_enabled => On => On datadog.trace.web_analytics_enabled => Off => Off datadog.web_analytics_enabled => Off => Off datadog.trace.web_analytics_sample_rate => 1 => 1 datadog.web_analytics_sample_rate => 1 => 1 datadog.trace.wordpress_enabled => On => On datadog.trace.wordpress_analytics_enabled => Off => Off datadog.wordpress_analytics_enabled => Off => Off datadog.trace.wordpress_analytics_sample_rate => 1 => 1 datadog.wordpress_analytics_sample_rate => 1 => 1 datadog.trace.yii_enabled => On => On datadog.trace.yii_analytics_enabled => Off => Off datadog.yii_analytics_enabled => Off => Off datadog.trace.yii_analytics_sample_rate => 1 => 1 datadog.yii_analytics_sample_rate => 1 => 1 datadog.trace.zendframework_enabled => On => On datadog.trace.zendframework_analytics_enabled => Off => Off datadog.zendframework_analytics_enabled => Off => Off datadog.trace.zendframework_analytics_sample_rate => 1 => 1 datadog.zendframework_analytics_sample_rate => 1 => 1 Extension 'datadog-profiling' not present.

Upgrading from

No response

Jan 15 '25 10:01 doctenahasib

Hello @doctenahasib,

Is your application running in a serverless environment? If yes, could you please set the DD_SPAWN_WORKER_USE_EXEC=1 environment variable. If no, could you please set DD_TRACE_LOG_FILE=/some/path/ddtrace.log and DD_TRACE_LOG_LEVEL=debug? There should be some information inside about the cause of the broken pipe then.

Jan 15 '25 19:01 bwoebi

@bwoebi Yes I am using AWS ECS Fargate.

For now the error disappeared after I did a new deployment. When the issue appears again I will try your solution.

Jan 16 '25 07:01 doctenahasib

@bwoebi It happened again. I will deploy a new release with the env variable DD_SPAWN_WORKER_USE_EXEC=1 and see if that helps.

Jan 16 '25 14:01 doctenahasib

@bwoebi DD_SPAWN_WORKER_USE_EXEC=1 did not solve the issue.

Jan 20 '25 07:01 doctenahasib

@doctenahasib Then, could you please install 1.6.3 and set DD_TRACE_LOG_FILE=/proc/1/fd/2 (this should make the log output of sidecar go to the container log) and DD_TRACE_DEBUG=1. This may contain useful information for getting to the root cause.

Jan 20 '25 11:01 bwoebi

@bwoebi I installed 1.6.3. I also enabled the debug logs but it generated so much logs that I had to disable it again because of cost. However what I found out is that when we have a spike in our cpu and/or memory than we get this datadog tracer issue otherwise not. Could it be that when the tracer crashes that it can not recover from itself ?

Feb 13 '25 07:02 doctenahasib

Our team has been running tracer v1.6.3 when we first started seeing this issue. Our dd agent was behind so we updated that to the latest version. We tried bumping resource requests, but no matter what the log comes back.

[ddtrace] [error] Failed sending remote config data: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

It can appear almost immediately or take as long as 40 minutes to appear, but it always come back.

Feb 14 '25 02:02 chrisbradleydev

Hey all,

I wanted to ask whether the issue still exists with 1.7.2? There have been quite a bunch of bugfixes by now on the sidecar side.

Mar 07 '25 14:03 bwoebi

We're still seeing the issue after upgrading to 1.7.2.

Our team has an ongoing support ticket with DD.

Mar 17 '25 19:03 chrisbradleydev

Hello We observed the following error:

[ddtrace] [error] Failed sending remote config data: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

on the version 1.10.0 (latest for now)

Jun 02 '25 12:06 vladislav-gregurco

We started working on mitigating this error. This needs some refactors though and will take a bit of time.

Jun 04 '25 11:06 bwoebi

Hi guys, is there any update for this issue? we are facing this same issue recently.

Jun 16 '25 03:06 luanvan95

We are seeing similar errors in our Laravel queue (supervisor + horizon - Laravel v11).

PHP 8.4.8 (cli) (built: Jun  6 2025 17:38:35) (NTS)
Copyright (c) The PHP Group
Built by https://github.com/docker-library/php
Zend Engine v4.4.8, Copyright (c) Zend Technologies
    with Zend OPcache v8.4.8, Copyright (c), by Zend Technologies
    with ddtrace v1.10.0, Copyright Datadog, by Datadog
    with ddappsec v1.10.0, Copyright Datadog, by Datadog

Work around?

Is the work around to disable tracing for the queue? We already had to disable the profiler because of segfault, now we have this issue with the tracer. Being new customer to DataDog, this is not a great experience with the APM support.

More Info

Pod configuration:

DD_TRACE_GENERATE_ROOT_SPAN: 0
DD_TRACE_AUTO_FLUSH_ENABLED: 1
DD_SERVICE: "xxx"
DD_ENV: "xxx"

At some point the pod completely crash with

[ddtrace] [error] Failed allocating shared memory: EMFILE: Too many open files

Which I'm guessing it's because it has too many errors that are stacking to be reported but cannot communicate with the sidecar.

[ddtrace] [error] Failed sending remote config data: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }
[ddtrace] [error] Failed allocating shared memory: EMFILE: Too many open files

PHP Fatal error:  Uncaught ErrorException: include(/var/www/vendor/symfony/string/UnicodeString.php): Failed to open stream: No file descriptors available in /var/www/vendor/composer/ClassLoader.php:576
Stack trace:
#0 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(256): Illuminate\Foundation\Bootstrap\HandleExceptions->handleError(2, 'include(/var/ww...', '/var/www/vendor...', 576)
#1 /var/www/vendor/composer/ClassLoader.php(576): Illuminate\Foundation\Bootstrap\HandleExceptions->{closure:Illuminate\Foundation\Bootstrap\HandleExceptions::forwardsTo():255}(2, 'include(/var/ww...', '/var/www/vendor...', 576)
#2 /var/www/vendor/composer/ClassLoader.php(576): include('/var/www/vendor...')
#3 /var/www/vendor/composer/ClassLoader.php(427): {closure:Composer\Autoload\ClassLoader::initializeIncludeClosure():575}('/var/www/vendor...')
#4 /var/www/vendor/symfony/console/Helper/Helper.php(45): Composer\Autoload\ClassLoader->loadClass('Symfony\\Compone...')
#5 /var/www/vendor/symfony/console/Application.php(874): Symfony\Component\Console\Helper\Helper::width('Uncaught ErrorE...')
#6 /var/www/vendor/symfony/console/Application.php(845): Symfony\Component\Console\Application->doRenderThrowable(Object(Symfony\Component\ErrorHandler\Error\FatalError), Object(Symfony\Component\Console\Output\ConsoleOutput))
#7 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Exceptions/Handler.php(1027): Symfony\Component\Console\Application->renderThrowable(Object(Symfony\Component\ErrorHandler\Error\FatalError), Object(Symfony\Component\Console\Output\ConsoleOutput))
#8 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(208): Illuminate\Foundation\Exceptions\Handler->renderForConsole(Object(Symfony\Component\Console\Output\ConsoleOutput), Object(Symfony\Component\ErrorHandler\Error\FatalError))
#9 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(190): Illuminate\Foundation\Bootstrap\HandleExceptions->renderForConsole(Object(Symfony\Component\ErrorHandler\Error\FatalError))
#10 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(232): Illuminate\Foundation\Bootstrap\HandleExceptions->handleException(Object(Symfony\Component\ErrorHandler\Error\FatalError))
#11 /var/www/vendor/laravel/framework/src/Illuminate/Foundation/Bootstrap/HandleExceptions.php(256): Illuminate\Foundation\Bootstrap\HandleExceptions->handleShutdown()
#12 [internal function]: Illuminate\Foundation\Bootstrap\HandleExceptions->{closure:Illuminate\Foundation\Bootstrap\HandleExceptions::forwardsTo():255}()
#13 {main}
  thrown in /var/www/vendor/composer/ClassLoader.php on line 576
[ddtrace] [error] Failed flushing telemetry buffer: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }
[ddtrace] [error] Failed flushing service data: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }
[ddtrace] [error] Failed signaling lifecycle end: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

Jul 02 '25 03:07 Maxwell2022

encountered this issue as well with out php 8.3 + laravel 11 + horizon + supervisorctl setup. The queues suddenly stop randomly and i see the same errors.

[21-Jul-2025 16:46:29 UTC] [ddtrace] [error] Failed sending remote config data: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }
[21-Jul-2025 16:46:29 UTC] [ddtrace] [error] Failed sending traces to the sidecar: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

I noticed this started when I started configuring remote configuration APM sampling. I tried updating from ddtrace 1.7.0 to 1.10.0 and it's still there. What fixed it for me was when I set datadog.remote_config_enabled = false. Also, it fixed our long time issue where php artisan tinker crashes into an endless loop due to the error Warning Trying to access array offset on null

Jul 23 '25 14:07 angeloinqbyte

dd-trace-php dd-trace-php copied to clipboard

[Bug]: Spamming LOG: [ddtrace] [error] Failed sending traces to the sidecar: Os { code: 32, kind: BrokenPipe, message: "Broken pipe" }

Bug report

PHP version

Tracer or profiler version

Installed extensions

Output of phpinfo()

Upgrading from

Work around?

More Info

dd-trace-php
dd-trace-php copied to clipboard

Output of `phpinfo()`