1.23.0 blocks when building Clojure uberjar and AOT is enabled

[antonmos]

when the datadog java agent 1.23.0+ is configured 'lein uberjar' freezes indefinitely. it works fine with 1.22.0 or before, OR when DD_INSTRUMENTATION_TELEMETRY_ENABLED=false

Minimal project to reproduce the issue is attached test-dd.zip

in the root of the project run lein uberjar. When dd-java-agent 1.23.0+, we see output like this

12:43:17:➜  test-dd lein uberjar
[dd.trace 2024-01-02 12:43:24:844 -0600] [dd-telemetry] WARN datadog.telemetry.TelemetryRouter - Got FAILURE sending telemetry request to http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry.
[dd.trace 2024-01-02 12:43:24:847 -0600] [dd-telemetry] INFO datadog.telemetry.TelemetryRouter - Agent Telemetry endpoint failed. Telemetry will be sent to Intake.
[dd.trace 2024-01-02 12:43:25:026 -0600] [dd-task-scheduler] INFO datadog.trace.agent.core.StatusLogger - DATADOG TRACER CONFIGURATION {"version":"1.23.0~94ed1d0fce","os_name":"Mac OS X","os_version":"10.16","architecture":"x86_64","lang":"jvm","lang_version":"11.0.10","jvm_vendor":"AdoptOpenJDK","jvm_version":"11.0.10+9","java_class_version":"55.0","http_nonProxyHosts":"null","http_proxyHost":"null","enabled":true,"service":"clojure.main","agent_url":"http://localhost:8126","agent_error":true,"debug":false,"trace_propagation_style_extract":["datadog"],"trace_propagation_style_inject":["datadog"],"analytics_enabled":false,"sampling_rules":[{},{}],"priority_sampling_enabled":true,"logs_correlation_enabled":true,"profiling_enabled":false,"remote_config_enabled":true,"debugger_enabled":false,"appsec_enabled":"ENABLED_INACTIVE","telemetry_enabled":true,"dd_version":"","health_checks_enabled":true,"configuration_file":"no config file present","runtime_id":"26ceda8d-5eb2-4512-9fbf-8d3ad4aa8068","logging_settings":{"levelInBrackets":false,"dateTimeFormat":"'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'","logFile":"System.err","configurationFile":"simplelogger.properties","showShortLogName":false,"showDateTime":true,"showLogName":true,"showThreadName":true,"defaultLogLevel":"INFO","warnLevelString":"WARN","embedException":false},"cws_enabled":false,"cws_tls_refresh":5000,"datadog_profiler_enabled":true,"datadog_profiler_safe":true,"datadog_profiler_enabled_overridden":false}
[dd.trace 2024-01-02 12:43:25:583 -0600] [dd-trace-processor] WARN datadog.trace.agent.common.writer.ddagent.DDAgentApi - Error while sending 1 (size=476B) traces. Total: 1, Received: 1, Sent: 0, Failed: 1. java.net.ConnectException: Failed to connect to localhost/0:0:0:0:0:0:0:1:8126 (Will not log errors for 5 minutes)
Compiling test-dd.core

and it blocks indefinitely.

When dd-java-agent 1.22.0-, we see output like this

12:44:29:➜  test-dd lein uberjar   
[dd.trace 2024-01-02 12:44:35:405 -0600] [dd-task-scheduler] INFO datadog.trace.agent.core.StatusLogger - DATADOG TRACER CONFIGURATION {"version":"1.22.0~848d3775b0","os_name":"Mac OS X","os_version":"10.16","architecture":"x86_64","lang":"jvm","lang_version":"11.0.10","jvm_vendor":"AdoptOpenJDK","jvm_version":"11.0.10+9","java_class_version":"55.0","http_nonProxyHosts":"null","http_proxyHost":"null","enabled":true,"service":"clojure.main","agent_url":"http://localhost:8126","agent_error":true,"debug":false,"trace_propagation_style_extract":["datadog"],"trace_propagation_style_inject":["datadog"],"analytics_enabled":false,"sampling_rules":[{},{}],"priority_sampling_enabled":true,"logs_correlation_enabled":true,"profiling_enabled":false,"remote_config_enabled":true,"debugger_enabled":false,"appsec_enabled":"ENABLED_INACTIVE","telemetry_enabled":true,"dd_version":"","health_checks_enabled":true,"configuration_file":"no config file present","runtime_id":"ee8e5012-daf3-44bb-bed2-8b411018e373","logging_settings":{"levelInBrackets":false,"dateTimeFormat":"'[dd.trace 'yyyy-MM-dd HH:mm:ss:SSS Z']'","logFile":"System.err","configurationFile":"simplelogger.properties","showShortLogName":false,"showDateTime":true,"showLogName":true,"showThreadName":true,"defaultLogLevel":"INFO","warnLevelString":"WARN","embedException":false},"cws_enabled":false,"cws_tls_refresh":5000,"datadog_profiler_enabled":true,"datadog_profiler_safe":true,"datadog_profiler_enabled_overridden":false}
[dd.trace 2024-01-02 12:44:35:729 -0600] [dd-trace-processor] WARN datadog.trace.agent.common.writer.ddagent.DDAgentApi - Error while sending 1 (size=476B) traces. Total: 1, Received: 1, Sent: 0, Failed: 1. java.net.ConnectException: Failed to connect to localhost/0:0:0:0:0:0:0:1:8126 (Will not log errors for 5 minutes)
Compiling test-dd.core
Created /private/tmp/test-dd/target/test-dd-0.1.0-SNAPSHOT.jar
Created /private/tmp/test-dd/target/test-dd-0.1.0-SNAPSHOT-standalone.jar
12:44:39:➜  test-dd 

Note that it finished in 10s.

It appears that the following logs may be relevant.

[dd.trace 2024-01-02 12:43:24:844 -0600] [dd-telemetry] WARN datadog.telemetry.TelemetryRouter - Got FAILURE sending telemetry request to http://localhost:8126/telemetry/proxy/api/v2/apmtelemetry.
[dd.trace 2024-01-02 12:43:24:847 -0600] [dd-telemetry] INFO datadog.telemetry.TelemetryRouter - Agent Telemetry endpoint failed. Telemetry will be sent to Intake.

Also, running DD_INSTRUMENTATION_TELEMETRY_ENABLED=false lein uberjar works correctly even with dd-java-agent 1.23+

[antonmos]

adding "-Ddd.instrumentation.telemetry.enabled=false" to :jvm-opts also works.

[ygree]

Thank you for bringing this to our attention. We were unable to reproduce this issue using your example. Can you please double check if the issue is still reproducible on your end?

[antonmos]

it is reproducible.

# lein version
Leiningen 2.10.0 on Java 17.0.7 OpenJDK 64-Bit Server VM

on macOS 13.6.3

[ygree]

Thanks for confirming it. We have found the source of issue and implemented fix in #6451