DataDog
[Bug]: Memory leak in creating Tracer
Bug report
Hi,
After an ugprade of the library, we are experiencing memory leaks.
We are running a long running process that starts tracing every time a message is consumed from Kafka and we close the global trace every time message is successfully processed.
This is the code we run every time we consume a message to start tracing
protected function startDatadogTracing(
string $topicName,
string $traceId = null,
string $parentId = null
): void {
if (!empty($traceId) && !empty($parentId)) {
set_distributed_tracing_context($traceId, $parentId);
}
$tracer = GlobalTracer::get();
GlobalTracer::set($tracer);
$resourceName = $topicName . '::process';
$options = ['start_time' => ((int)(microtime(true) * 1000 * 1000))];
$rootSpan = $tracer->startRootSpan(
'Jobs',
$options
)
->getSpan();
$rootSpan->setTag(Tag::SPAN_TYPE, Type::CLI);
$rootSpan->setTag(Tag::SERVICE_NAME, 'broker.xxx');
$rootSpan->setTag(Tag::RESOURCE_NAME, $resourceName, true);
$rootSpan->setTag('command_origin', $this->signature);
if (!$this->isTracingStarted()) {
$this->setupTracingOfClasses();
$this->setTracingStarted(true);
}
}
and this is the whole code together with an example how we used to close root spans and send them to dd-agent./
$this->startDatadogTracing(
$message->topic_name . '-persist-offer-created',
$message->headers['x-datadog-trace-id'] ?? null,
$message->headers['x-datadog-parent-id'] ?? null
);
$this->offerService->processOfferCreatedToPersistenceCommand(
$message
);
GlobalTracer::get()->getRootScope()?->getSpan()->setTag('result', 'OK');
GlobalTracer::get()->getRootScope()?->close();
GlobalTracer::get()->flush()
however when running the setup locally, after every processed message memory increases by a bit. I am tracking the used memory using this simple line of code
dump(memory_get_usage());
Do you guys have any idea what we might be doing wrong?
Thank you.
PHP version
8.1.27
Tracer or profiler version
0.98.0
Installed extensions
/var/www/app # php -m [PHP Modules] bcmath calendar Core ctype curl datadog-profiling date ddappsec ddtrace dom fileinfo filter ftp hash iconv intl json libxml mbstring mysqlnd openssl pcntl pcre PDO pdo_mysql pdo_sqlite Phar posix rdkafka readline redis Reflection session SimpleXML soap sockets sodium SPL sqlite3 standard tokenizer xdebug xml xmlreader xmlwriter Zend OPcache zip zlib
[Zend Modules] Xdebug Zend OPcache datadog-profiling ddappsec ddtrace
Output of phpinfo()
/var/www/app # php --ri=ddtrace; php --ri=datadog-profiling
ddtrace
Datadog PHP tracer extension For help, check out the documentation at https://docs.datadoghq.com/tracing/languages/php/ (c) Datadog 2020
Datadog tracing support => enabled Version => 0.98.0 DATADOG TRACER CONFIGURATION => { "date": "2024-03-25T12:58:26Z", "os_name": "Linux 1bb3197e090f 5.15.49-linuxkit-pr #1 SMP PREEMPT Thu May 25 07:27:39 UTC 2023 aarch64", "os_version": "5.15.49-linuxkit-pr", "version": "0.98.0", "lang": "php", "lang_version": "8.1.26", "env": null, "enabled": true, "service": "broker.xxx", "enabled_cli": true, "agent_url": "http://datadog:8126", "debug": false, "analytics_enabled": false, "sample_rate": -1, "sampling_rules": [], "tags": [], "service_mapping": { "pdo": "broker.xxx.db", "phpredis": "broker.xxx.redis" }, "distributed_tracing_enabled": true, "priority_sampling_enabled": true, "dd_version": null, "architecture": "aarch64", "sapi": "cli", "datadog.trace.request_init_hook": "/opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php", "open_basedir_configured": false, "uri_fragment_regex": null, "uri_mapping_incoming": null, "uri_mapping_outgoing": null, "auto_flush_enabled": true, "generate_root_span": false, "http_client_split_by_domain": false, "measure_compile_time": true, "report_hostname_on_root_span": false, "traced_internal_functions": null, "auto_prepend_file_configured": false, "integrations_disabled": "default", "enabled_from_env": true, "opcache.file_cache": null, "agent_error": "Could not resolve host: datadog" }
Diagnostics
agent_error => Could not resolve host: datadog Diagnostic checks => failed
Directive => Local Value => Master Value ddtrace.disable => Off => Off ddtrace.cgroup_file => /proc/self/cgroup => /proc/self/cgroup datadog.trace.sidecar_trace_sender => Off => Off datadog.trace.request_init_hook => /opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php ddtrace.request_init_hook => /opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php => /opt/datadog/dd-library/0.98.0/dd-trace-sources/bridge/dd_wrap_autoloader.php datadog.trace.agent_url => no value => no value datadog.agent_host => datadog => datadog datadog.dogstatsd_url => no value => no value datadog.api_key => no value => no value datadog.distributed_tracing => On => On datadog.dogstatsd_port => 8125 => 8125 datadog.env => no value => no value datadog.autofinish_spans => Off => Off datadog.trace.url_as_resource_names_enabled => On => On datadog.http_server_route_based_naming => On => On datadog.integrations_disabled => default => default datadog.priority_sampling => On => On datadog.service => broker.offer-service => broker.offer-service datadog.service_name => broker.offer-service => broker.offer-service datadog.service_mapping => pdo:broker.xxx.db,phpredis:broker.xxx.redis => pdo:broker.xxx.db,phpredis:broker.xxx.redis datadog.tags => no value => no value datadog.trace.global_tags => no value => no value datadog.trace.agent_port => 8126 => 8126 datadog.trace.analytics_enabled => Off => Off datadog.trace.append_trace_ids_to_logs => Off => Off datadog.trace.auto_flush_enabled => On => On datadog.trace.cli_enabled => On => On datadog.trace.measure_compile_time => On => On datadog.trace.debug => Off => Off datadog.trace.enabled => On => On datadog.instrumentation_telemetry_enabled => On => On datadog.trace.health_metrics_enabled => Off => Off datadog.trace.health_metrics_heartbeat_sample_rate => 0.001 => 0.001 datadog.trace.db_client_split_by_instance => Off => Off datadog.trace.http_client_split_by_domain => Off => Off datadog.trace.redis_client_split_by_host => On => On datadog.trace.memory_limit => no value => no value datadog.trace.report_hostname => Off => Off datadog.trace.flush_collect_cycles => Off => Off datadog.trace.laravel_queue_distributed_tracing => On => On datadog.trace.remove_root_span_laravel_queue => On => On datadog.trace.remove_autoinstrumentation_orphans => Off => Off datadog.trace.resource_uri_fragment_regex => no value => no value datadog.trace.resource_uri_mapping_incoming => no value => no value datadog.trace.resource_uri_mapping_outgoing => no value => no value datadog.trace.resource_uri_query_param_allowed => no value => no value datadog.trace.http_url_query_param_allowed => * => * datadog.trace.http_post_data_param_allowed => no value => no value datadog.trace.rate_limit => 0 => 0 datadog.trace.sample_rate => -1 => -1 datadog.sampling_rate => -1 => -1 datadog.trace.sampling_rules => [] => [] datadog.trace.sampling_rules_format => regex => regex datadog.span_sampling_rules => [] => [] datadog.span_sampling_rules_file => no value => no value datadog.trace.header_tags => no value => no value datadog.trace.x_datadog_tags_max_length => 512 => 512 datadog.trace.peer_service_mapping => no value => no value datadog.trace.peer_service_defaults_enabled => Off => Off datadog.trace.remove_integration_service_names_enabled => Off => Off datadog.trace.propagate_service => Off => Off datadog.trace.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header datadog.propagation_style_extract => datadog,tracecontext,B3,B3 single header => datadog,tracecontext,B3,B3 single header datadog.trace.propagation_style_inject => datadog,tracecontext => datadog,tracecontext datadog.propagation_style_inject => datadog,tracecontext => datadog,tracecontext datadog.trace.propagation_style => datadog,tracecontext => datadog,tracecontext datadog.trace.traced_internal_functions => no value => no value datadog.trace.agent_timeout => 500 => 500 datadog.trace.agent_connect_timeout => 100 => 100 datadog.trace.debug_prng_seed => -1 => -1 datadog.log_backtrace => Off => Off datadog.trace.generate_root_span => Off => Off datadog.trace.spans_limit => 5000 => 5000 datadog.trace.128_bit_traceid_generation_enabled => On => On datadog.trace.128_bit_traceid_logging_enabled => Off => Off datadog.trace.agent_max_consecutive_failures => 3 => 3 datadog.trace.agent_attempt_retry_time_msec => 5000 => 5000 datadog.trace.bgs_connect_timeout => 2000 => 2000 datadog.trace.bgs_timeout => 5000 => 5000 datadog.trace.agent_flush_interval => 5000 => 5000 datadog.trace.agent_flush_after_n_requests => 10 => 10 datadog.trace.shutdown_timeout => 5000 => 5000 datadog.trace.startup_logs => On => On datadog.trace.once_logs => On => On datadog.trace.agent_retries => 0 => 0 datadog.trace.agent_debug_verbose_curl => Off => Off datadog.trace.debug_curl_output => Off => Off datadog.trace.beta_high_memory_pressure_percent => 80 => 80 datadog.trace.agentless => Off => Off datadog.trace.warn_legacy_dd_trace => On => On datadog.trace.retain_thread_capabilities => Off => Off datadog.version => no value => no value datadog.trace.obfuscation_query_string_regexp => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9.-]+)?) => (?i)(?:(?:"|%22)?)(?:(?:old[-]?|new[-]?)?p(?:ass)?w(?:or)?d(?:1|2)?|pass(?:[-]?phrase)?|secret|(?:api[-]?|private[-]?|public[-]?|access[-]?|secret[-]?|app(?:lication)?[-]?)key(?:[-]?id)?|token|consumer[-]?(?:id|key|secret)|sign(?:ed|ature)?|auth(?:entication|orization)?)(?:(?:\s|%20)(?:=|%3D)[^&]+|(?:"|%22)(?:\s|%20)(?::|%3A)(?:\s|%20)*(?:"|%22)(?:%2[^2]|%[^2]|[^"%])+(?:"|%22))|(?:bearer(?:\s|%20)+[a-z0-9.-]+|token(?::|%3A)[a-z0-9]{13}|gh[opsu][0-9a-zA-Z]{36}|eyI-L+.eyI-L+(?:.(?:[\w.+/=-]|%3D|%2F|%2B)+)?|-{5}BEGIN(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY-{5}[^-]+-{5}END(?:[a-z\s]|%20)+PRIVATE(?:\s|%20)KEY(?:-{5})?(?:\n|%0A)?|(?:ssh-(?:rsa|dss)|ecdsa-[a-z0-9]+-[a-z0-9]+)(?:\s|%20|%09)+(?:[a-z0-9/.+]|%2F|%5C|%2B){100,}(?:=|%3D)*(?:(?:\s|%20|%09)+[a-z0-9.-]+)?) datadog.trace.client_ip_enabled => Off => Off datadog.trace.client_ip_header => no value => no value datadog.trace.forked_process => On => On datadog.trace.hook_limit => 100 => 100 datadog.trace.agent_max_payload_size => 52428800 => 52428800 datadog.trace.agent_stack_initial_size => 131072 => 131072 datadog.trace.agent_stack_backlog => 12 => 12 datadog.trace.propagate_user_id_default => Off => Off datadog.dbm_propagation_mode => full => full datadog.trace.wordpress_additional_actions => no value => no value datadog.trace.wordpress_callbacks => Off => Off datadog.trace.wordpress_enhanced_integration => Off => Off datadog.trace.otel_enabled => Off => Off datadog.trace.log_file => no value => no value datadog.trace.log_level => error => error datadog.trace.amqp_enabled => On => On datadog.trace.amqp_analytics_enabled => Off => Off datadog.amqp_analytics_enabled => Off => Off datadog.trace.amqp_analytics_sample_rate => 1 => 1 datadog.amqp_analytics_sample_rate => 1 => 1 datadog.trace.cakephp_enabled => On => On datadog.trace.cakephp_analytics_enabled => Off => Off datadog.cakephp_analytics_enabled => Off => Off datadog.trace.cakephp_analytics_sample_rate => 1 => 1 datadog.cakephp_analytics_sample_rate => 1 => 1 datadog.trace.codeigniter_enabled => On => On datadog.trace.codeigniter_analytics_enabled => Off => Off datadog.codeigniter_analytics_enabled => Off => Off datadog.trace.codeigniter_analytics_sample_rate => 1 => 1 datadog.codeigniter_analytics_sample_rate => 1 => 1 datadog.trace.exec_enabled => On => On datadog.trace.exec_analytics_enabled => Off => Off datadog.exec_analytics_enabled => Off => Off datadog.trace.exec_analytics_sample_rate => 1 => 1 datadog.exec_analytics_sample_rate => 1 => 1 datadog.trace.curl_enabled => On => On datadog.trace.curl_analytics_enabled => Off => Off datadog.curl_analytics_enabled => Off => Off datadog.trace.curl_analytics_sample_rate => 1 => 1 datadog.curl_analytics_sample_rate => 1 => 1 datadog.trace.drupal_enabled => On => On datadog.trace.drupal_analytics_enabled => Off => Off datadog.drupal_analytics_enabled => Off => Off datadog.trace.drupal_analytics_sample_rate => 1 => 1 datadog.drupal_analytics_sample_rate => 1 => 1 datadog.trace.elasticsearch_enabled => On => On datadog.trace.elasticsearch_analytics_enabled => Off => Off datadog.elasticsearch_analytics_enabled => Off => Off datadog.trace.elasticsearch_analytics_sample_rate => 1 => 1 datadog.elasticsearch_analytics_sample_rate => 1 => 1 datadog.trace.eloquent_enabled => On => On datadog.trace.eloquent_analytics_enabled => Off => Off datadog.eloquent_analytics_enabled => Off => Off datadog.trace.eloquent_analytics_sample_rate => 1 => 1 datadog.eloquent_analytics_sample_rate => 1 => 1 datadog.trace.guzzle_enabled => On => On datadog.trace.guzzle_analytics_enabled => Off => Off datadog.guzzle_analytics_enabled => Off => Off datadog.trace.guzzle_analytics_sample_rate => 1 => 1 datadog.guzzle_analytics_sample_rate => 1 => 1 datadog.trace.laminas_enabled => On => On datadog.trace.laminas_analytics_enabled => Off => Off datadog.laminas_analytics_enabled => Off => Off datadog.trace.laminas_analytics_sample_rate => 1 => 1 datadog.laminas_analytics_sample_rate => 1 => 1 datadog.trace.laravel_enabled => On => On datadog.trace.laravel_analytics_enabled => Off => Off datadog.laravel_analytics_enabled => Off => Off datadog.trace.laravel_analytics_sample_rate => 1 => 1 datadog.laravel_analytics_sample_rate => 1 => 1 datadog.trace.laravelqueue_enabled => On => On datadog.trace.laravelqueue_analytics_enabled => Off => Off datadog.laravelqueue_analytics_enabled => Off => Off datadog.trace.laravelqueue_analytics_sample_rate => 1 => 1 datadog.laravelqueue_analytics_sample_rate => 1 => 1 datadog.trace.logs_enabled => Off => Off datadog.logs_injection => Off => Off datadog.trace.logs_analytics_enabled => Off => Off datadog.logs_analytics_enabled => Off => Off datadog.trace.logs_analytics_sample_rate => 1 => 1 datadog.logs_analytics_sample_rate => 1 => 1 datadog.trace.lumen_enabled => On => On datadog.trace.lumen_analytics_enabled => Off => Off datadog.lumen_analytics_enabled => Off => Off datadog.trace.lumen_analytics_sample_rate => 1 => 1 datadog.lumen_analytics_sample_rate => 1 => 1 datadog.trace.magento_enabled => On => On datadog.trace.magento_analytics_enabled => Off => Off datadog.magento_analytics_enabled => Off => Off datadog.trace.magento_analytics_sample_rate => 1 => 1 datadog.magento_analytics_sample_rate => 1 => 1 datadog.trace.memcache_enabled => On => On datadog.trace.memcache_analytics_enabled => Off => Off datadog.memcache_analytics_enabled => Off => Off datadog.trace.memcache_analytics_sample_rate => 1 => 1 datadog.memcache_analytics_sample_rate => 1 => 1 datadog.trace.memcached_enabled => On => On datadog.trace.memcached_analytics_enabled => Off => Off datadog.memcached_analytics_enabled => Off => Off datadog.trace.memcached_analytics_sample_rate => 1 => 1 datadog.memcached_analytics_sample_rate => 1 => 1 datadog.trace.mongo_enabled => On => On datadog.trace.mongo_analytics_enabled => Off => Off datadog.mongo_analytics_enabled => Off => Off datadog.trace.mongo_analytics_sample_rate => 1 => 1 datadog.mongo_analytics_sample_rate => 1 => 1 datadog.trace.mongodb_enabled => On => On datadog.trace.mongodb_analytics_enabled => Off => Off datadog.mongodb_analytics_enabled => Off => Off datadog.trace.mongodb_analytics_sample_rate => 1 => 1 datadog.mongodb_analytics_sample_rate => 1 => 1 datadog.trace.mysqli_enabled => On => On datadog.trace.mysqli_analytics_enabled => Off => Off datadog.mysqli_analytics_enabled => Off => Off datadog.trace.mysqli_analytics_sample_rate => 1 => 1 datadog.mysqli_analytics_sample_rate => 1 => 1 datadog.trace.nette_enabled => On => On datadog.trace.nette_analytics_enabled => Off => Off datadog.nette_analytics_enabled => Off => Off datadog.trace.nette_analytics_sample_rate => 1 => 1 datadog.nette_analytics_sample_rate => 1 => 1 datadog.trace.pcntl_enabled => On => On datadog.trace.pcntl_analytics_enabled => Off => Off datadog.pcntl_analytics_enabled => Off => Off datadog.trace.pcntl_analytics_sample_rate => 1 => 1 datadog.pcntl_analytics_sample_rate => 1 => 1 datadog.trace.pdo_enabled => On => On datadog.trace.pdo_analytics_enabled => Off => Off datadog.pdo_analytics_enabled => Off => Off datadog.trace.pdo_analytics_sample_rate => 1 => 1 datadog.pdo_analytics_sample_rate => 1 => 1 datadog.trace.phpredis_enabled => On => On datadog.trace.phpredis_analytics_enabled => Off => Off datadog.phpredis_analytics_enabled => Off => Off datadog.trace.phpredis_analytics_sample_rate => 1 => 1 datadog.phpredis_analytics_sample_rate => 1 => 1 datadog.trace.predis_enabled => On => On datadog.trace.predis_analytics_enabled => Off => Off datadog.predis_analytics_enabled => Off => Off datadog.trace.predis_analytics_sample_rate => 1 => 1 datadog.predis_analytics_sample_rate => 1 => 1 datadog.trace.psr18_enabled => On => On datadog.trace.psr18_analytics_enabled => Off => Off datadog.psr18_analytics_enabled => Off => Off datadog.trace.psr18_analytics_sample_rate => 1 => 1 datadog.psr18_analytics_sample_rate => 1 => 1 datadog.trace.roadrunner_enabled => On => On datadog.trace.roadrunner_analytics_enabled => Off => Off datadog.roadrunner_analytics_enabled => Off => Off datadog.trace.roadrunner_analytics_sample_rate => 1 => 1 datadog.roadrunner_analytics_sample_rate => 1 => 1 datadog.trace.sqlsrv_enabled => On => On datadog.trace.sqlsrv_analytics_enabled => Off => Off datadog.sqlsrv_analytics_enabled => Off => Off datadog.trace.sqlsrv_analytics_sample_rate => 1 => 1 datadog.sqlsrv_analytics_sample_rate => 1 => 1 datadog.trace.slim_enabled => On => On datadog.trace.slim_analytics_enabled => Off => Off datadog.slim_analytics_enabled => Off => Off datadog.trace.slim_analytics_sample_rate => 1 => 1 datadog.slim_analytics_sample_rate => 1 => 1 datadog.trace.symfony_enabled => On => On datadog.trace.symfony_analytics_enabled => Off => Off datadog.symfony_analytics_enabled => Off => Off datadog.trace.symfony_analytics_sample_rate => 1 => 1 datadog.symfony_analytics_sample_rate => 1 => 1 datadog.trace.web_enabled => On => On datadog.trace.web_analytics_enabled => Off => Off datadog.web_analytics_enabled => Off => Off datadog.trace.web_analytics_sample_rate => 1 => 1 datadog.web_analytics_sample_rate => 1 => 1 datadog.trace.wordpress_enabled => On => On datadog.trace.wordpress_analytics_enabled => Off => Off datadog.wordpress_analytics_enabled => Off => Off datadog.trace.wordpress_analytics_sample_rate => 1 => 1 datadog.wordpress_analytics_sample_rate => 1 => 1 datadog.trace.yii_enabled => On => On datadog.trace.yii_analytics_enabled => Off => Off datadog.yii_analytics_enabled => Off => Off datadog.trace.yii_analytics_sample_rate => 1 => 1 datadog.yii_analytics_sample_rate => 1 => 1 datadog.trace.zendframework_enabled => On => On datadog.trace.zendframework_analytics_enabled => Off => Off datadog.zendframework_analytics_enabled => Off => Off datadog.trace.zendframework_analytics_sample_rate => 1 => 1 datadog.zendframework_analytics_sample_rate => 1 => 1
datadog-profiling
Version => 0.98.0
Profiling Enabled => false
Profiling Experimental Features Enabled => false (profiling disabled)
Experimental CPU Time Profiling Enabled => false (profiling disabled)
Allocation Profiling Enabled => false (profiling disabled)
Timeline Enabled => false (profiling disabled)
Exception Profiling Enabled => false (profiling disabled)
Endpoint Collection Enabled => false (profiling disabled)
Platform's CPU Time API Works => true
Profiling Log Level => off (profiling disabled)
Profiling Agent Endpoint => http://datadog:8126/
Application's Environment (DD_ENV) =>
Application's Service (DD_SERVICE) => broker.xxx
Application's Version (DD_VERSION) =>
Directive => Local Value => Master Value datadog.profiling.enabled => Off => Off datadog.profiling.experimental_features_enabled => Off => Off datadog.profiling.endpoint_collection_enabled => On => On datadog.profiling.experimental_cpu_time_enabled => On => On datadog.profiling.experimental_cpu_enabled => On => On datadog.profiling.allocation_enabled => Off => Off datadog.profiling.experimental_allocation_enabled => Off => Off datadog.profiling.timeline_enabled => On => On datadog.profiling.experimental_timeline_enabled => On => On datadog.profiling.exception_enabled => Off => Off datadog.profiling.experimental_exception_enabled => Off => Off datadog.profiling.exception_message_enabled => Off => Off datadog.profiling.exception_sampling_distance => 100 => 100 datadog.profiling.experimental_exception_sampling_distance => 100 => 100 datadog.profiling.log_level => off => off datadog.profiling.output_pprof => no value => no value
Upgrading from
Upgrading from 0.84.0 -> 0.98.0
Oh,
after exchanging
$tracer = GlobalTracer::get();
to
$tracer = new Tracer();
memory doesn't increase anymore. I am trying to think now whether it makes sense or not 🤔
Hey @iNviNho,
Is there something in the internal state of GlobalTracer::get() which is ever-growing? I.e. observing var_dump(GlobalTracer::get()), do you observe some arrays growing or data structures getting more and more nested?
After we call
GlobalTracer::get()->getRootScope()?->getSpan()->setTag('result', 'OK');
GlobalTracer::get()->getRootScope()?->close();
dump(GlobalTracer::get());
we always get DDTrace\Tracer object with a different object id as with every new message we consume, we create a new Tracer.
DDTrace\Tracer^ {#4063 // app/Console/Commands/LinkMultiSwapTradesCommand.php:91
-traces: []
-transport: DDTrace\Transport\Internal^ {#4087}
-propagators: array:2 [
"text_map" => DDTrace\Propagators\TextMap^ {#4086
-tracer: DDTrace\Tracer^ {#4063}
}
"http_headers" => DDTrace\Propagators\TextMap^ {#4086}
]
-config: array:3 [
"service_name" => "cli"
"enabled" => true
"global_tags" => []
]
-scopeManager: DDTrace\ScopeManager^ {#4037
-scopes: []
-hostRootScopes: array:1 [
0 => DDTrace\Scope^ {#4047
-span: DDTrace\Span^ {#4049
+context: DDTrace\SpanContext^ {#4052
+traceId: "89806696424059378"
+spanId: "89806696424059378"
+parentId: null
+isDistributedTracingActivationContext: false
+propagatedPrioritySampling: null
+origin: null
+parentContext: null
+baggageItems: []
}
+hasError: false
#internalSpan: ?
+internalSpan: DDTrace\RootSpanData {#4048
+name: "Jobs"
+resource: "..."
+service: "..."
+type: "cli"
+meta: array:5 [
"runtime-id" => "52936d31-37bd-414e-a474-8a0842804c58"
"version" => "0.0.1"
"command_origin" => "..."
"result" => "OK"
"_dd.p.dm" => "-0"
]
+metrics: array:2 [
"process_id" => 753.0
"_dd.agent_psr" => 1.0
]
+exception: null
+id: "89806696424059378"
+links: []
+peerServiceSources: []
+parent: null
+stack: DDTrace\SpanStack {#4006
+parent: DDTrace\SpanStack {#1
+parent: null
+active: null
}
+active: null
}
+origin: ? string
+propagatedTags: []
+samplingPriority: 1
+propagatedSamplingPriority: ? int
+tracestate: ? string
+tracestateTags: []
+parentId: ? string
+traceId: "6602ef6300000000013f0eb6eeceb1f2"
}
}
-scopeManager: DDTrace\ScopeManager^ {#4037}
-finishSpanOnClose: true
}
]
-rootContext: null
}
-serviceVersion: "0.0.1"
-environment: ""
-rootContext: null
}
and the memory keeps increasing until we do
gc_collect_cycles();
gc_mem_caches();
We see that all our daemon deployments slowly climb in memory usage. This screenshot shows a datadog profile of total allocated memory over an hour in one of our service.
Hey @TheLevti 👋
your problem looks like a different problem then what the author wrote. Could you please signup to our public Slack or open a Support Case with us? If you choose Slack, please ping "Florian Engelhardt" and/or "Bob Weinand"
Actually it is related as we investigate the same issue with @iNviNho.
What we have found out is that calling $tracer->reset() after flushing is fixing the memory leak.
Can we maybe have more details about when reset() should be called and whether thats the correct approach?
Why is the tracer accumulating already closed/flushed traces if reset is not called?
This call also allows us to use the global tracer instance as a singleton and not reinstantiate it after each trace.
@TheLevti unrelated question, but what tool do you use to measure memory usage and plot such graphs?
@TheLevti unrelated question, but what tool do you use to measure memory usage and plot such graphs?
Its datadog's profiler component. Shows you CPU time, wall time, memory allocations etc.