Datadog agent errors

[shanmukha511]

Describe what happened:

I have enabled APM for my applications and i don't see APM services nor traces getting reflected in datadog UI but i see some weird errors from application pods.Below errors.

**[dd.trace 2022-04-20 07:30:53:776 +0000] [OkHttp http://localhost:8126/...] WARN com.datadog.profiling.uploader.ProfileUploader - Failed to upload profile to http://localhost:8126/profiling/v1/input java.io.IOException: Permission denied

[dd.trace 2022-04-20 07:28:22:788 +0000] [dd-trace-processor] WARN datadog.trace.agent.common.writer.ddagent.DDAgentApi - Error while sending 2 (size=1KB) traces to the DD agent. Total: 108104, Received: 108104, Sent: 0, Failed: 108104. java.io.IOException: Permission denied (Will not log errors for 5 minutes)

[dd.trace 2022-04-20 07:27:46:967 +0000] [StatsD-Processor-1] WARN datadog.communication.monitor.DDAgentStatsDConnection - BufferOverflowException in StatsD client - /var/run/datadog/dsd.socket java.nio.BufferOverflowException: null (Will not log errors for 5 minutes)**

Describe what you expected:

The APM services and traces and profiles has to be reflected in Datadog APM section.

Steps to reproduce the issue: Have enabled APM for datadog using values.yaml and updated environment variables for application manifests

Additional environment details (Operating System, Cloud provider, etc): GKE

[mgvazquez]

Hi!, same error here. @shanmukha511, were you able to find some workaround?

[shanmukha511]

@mgvazquez nope

[veyselsahin]

same error here

[gfoligna-nyshex]

Same here

[s7an-it]

[dd.trace 2022-10-07 04:33:41:044 +0000] [StatsD-Processor-1] WARN datadog.communication.monitor.DDAgentStatsDConnection - BufferOverflowException in StatsD client - /var/run/datadog/dsd.socket java.nio.BufferOverflowException: null (Will not log errors for 5 minutes)

[avbhandaru]

Same here.

[sleshJdev]

Same here

[Demonca13]

Guys, has anyone figured out the problem?

[comstering]

Same here.

And I have another exception log at java.io.Exception.

[dd.trace 2023-01-31 19:12:57:613 +0000] [OkHttp {url}/...] WARN com.datadog.profiling.uploader.ProfileUploader - Failed to upload profile to {url}/profiling/v1/input (Will not log errors for 5 minutes)

java.io.IOException: unexpected end of stream on Connection{{url}, proxy=DIRECT hostAddress=/{url} cipherSuite=none protocol=http/1.1}

Has not been fixed yet?

[memoer]

Same here i have another "Will not log errors for 5 minutes" [dd.trace 2023-04-26 10:29:09:317 +0900] [OkHttp http://localhost:8126/...] WARN com.datadog.profiling.uploader.ProfileUploader - Failed to upload profile to http://localhost:8126/profiling/v1/input java.net.ConnectException: Failed to connect to localhost/127.0.0.1:8126 (Will not log errors for 5 minutes)

[doniyor2109]

@ahmed-mez @paulcacheux Can you help here?

We are have some of those logs as well. Not sure what is the reason of them. Could not find resources where this error was mentioned.

[ahmed-mez]

Hi all, sorry for the inconvenience!

These logs from the java library indicate that the tracer and profiler couldn't reach the agent, this is likely because of a misconfiguration in your app and/or the agent. Are you sure you have the env vars DD_TRACE_AGENT_URL or DD_AGENT_HOST configured correctly in your app?

Depending on your environment, please make sure you follow the steps here or the in-app guide to configure the connection between your app and the agent.

If you run on Kubernetes, you can leverage the Datadog Admission Controller to automate the configuration or even auto inject the java library without code change.

If the problem persists, please contact our support team to help you troubleshoot as it might also be specific to the network configurations in your infrastructure! In that case, we will likely need more information from you to help us investigate further.

Thank you!