claude-code icon indicating copy to clipboard operation
claude-code copied to clipboard

OpenTelemetry Monitoring: No Telemetry Data Exported

Open dzungtr opened this issue 9 months ago • 3 comments

Bug Description claude code OTEL monitor not working. I have set the env configuration according to the doc. However, my observability backend does not receive any data from claude. Enable debug or run command /status does not give any feedback.

Environment Info

  • Platform: linux
  • Terminal: Tabby
  • Version: 1.0.16
  • Feedback ID: 93ed3bb8-bf6e-4338-a4db-781daaba6c04

$HOME/.claude/settings.json

{
  "env": {
    "CLAUDE_CODE_ENABLE_TELEMETRY": "1",
    "OTEL_METRICS_EXPORTER": "otlp",
    "OTEL_EXPORTER_OTLP_PROTOCOL": "http",
    "OTEL_EXPORTER_OTLP_ENDPOINT": "http://localhost:4318"
  }
}

Otel collector manifest

apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel
spec:
  managementState: managed
  mode: deployment
  replicas: 1
  image: otel/opentelemetry-collector-contrib:latest
  podSecurityContext:
    runAsUser: 0
    runAsNonRoot: false
    fsGroup: 0
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      memory_limiter:
        check_interval: 1s
        limit_percentage: 75
        spike_limit_percentage: 15
      batch:
        send_batch_size: 10000
        timeout: 10s
    exporters:
      otlp/tempo:
        endpoint: tempo.observability:4317
        tls:
          insecure: true
      prometheusremotewrite:
        endpoint: http://kube-prom-stack-kube-prome-prometheus.observability:9090/api/v1/write
        tls:
          insecure: true
      loki:
        endpoint: http://loki.observability:3100/loki/api/v1/push
        default_labels_enabled:
          exporter: true
          job: true
          instance: true
          level: true
      debug:
        verbosity: detailed
        sampling_initial: 5
        sampling_thereafter: 200
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [otlp/tempo]
        metrics:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [prometheusremotewrite]
        logs:
          receivers: [otlp]
          processors: [memory_limiter, batch]
          exporters: [loki]
  ports:
    - name: otlp-grpc
      port: 4317
      protocol: TCP
    - name: otlp-http
      port: 4318
      protocol: TCP

otel-collector logs

2025-06-06T13:43:35.328Z        info    [email protected]/service.go:199 Setting up own telemetry...     {"resource": {}}
2025-06-06T13:43:35.328Z        info    [email protected]/exporter.go:178  starting prometheus remote write exporter {"resource": {}, "otelcol.component.id": "prometheusremotewrite", "otelcol.component.kind": "exporter", "otelcol.signal": "metrics", "ProtoMsg": "prometheus.WriteRequest"}
2025-06-06T13:43:35.329Z        info    [email protected]/memorylimiter.go:149     Using percentage memory limiter {"resource": {}, "otelcol.component.kind": "processor", "total_memory_mib": 61881, "limit_percentage": 75, "spike_limit_percentage": 15}
2025-06-06T13:43:35.329Z        info    [email protected]/memorylimiter.go:74      Memory limiter configured       {"resource": {}, "otelcol.component.kind": "processor", "limit_mib": 46411, "spike_limit_mib": 9282, "check_interval": 1}
2025-06-06T13:43:35.329Z        info    builders/builders.go:26 Deprecated component. Will be removed in future releases.       {"resource": {}, "otelcol.component.id": "loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-06-06T13:43:35.329Z        info    [email protected]/exporter.go:44    using the new Loki exporter     {"resource": {}, "otelcol.component.id": "loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs"}
2025-06-06T13:43:35.330Z        info    [email protected]/service.go:266 Starting otelcol-contrib...     {"resource": {}, "Version": "0.127.0", "NumCPU": 16}
2025-06-06T13:43:35.330Z        info    extensions/extensions.go:41     Starting extensions...  {"resource": {}}
2025-06-06T13:43:35.330Z        info    [email protected]/otlp.go:116       Starting GRPC server    {"resource": {}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "0.0.0.0:4317"}
2025-06-06T13:43:35.330Z        info    [email protected]/otlp.go:173       Starting HTTP server    {"resource": {}, "otelcol.component.id": "otlp", "otelcol.component.kind": "receiver", "endpoint": "0.0.0.0:4318"}
2025-06-06T13:43:35.330Z        info    [email protected]/service.go:289 Everything is ready. Begin running and processing data. {"resource": {}}

Errors

[]

dzungtr avatar Jun 06 '25 14:06 dzungtr

When was this added?

michael-hedgineer avatar Jun 06 '25 17:06 michael-hedgineer

I was able to get it working now. The issue lie in the problem with my config for claude in $HOME/.claude/settings.json.

Instead of

"OTEL_EXPORTER_OTLP_PROTOCOL": "http",

It should be

"OTEL_EXPORTER_OTLP_PROTOCOL": "http/json", # or "http/protobuf"

Beside, there was problem with my observability set up. Otel-collector did not push data to Prometheus properly. Both of the silent errors and very hard to tell which part is wrong. Whoever come to this thread later, I recommend using some well packages observability stack like signoz. It work out of the box. It took me a day to try export with different method and console to debug the problem. Happy spending for AI :)

dzungtr avatar Jun 07 '25 13:06 dzungtr

Hell yeah Tran! Would love to see what metrics are available and how to collect! Would make for an awesome post, or just reply to this issue.

Already DM'd you but I for one would be interested.


From: Dzung Tran (Tony) @.> Sent: Saturday, June 7, 2025 9:16 AM To: anthropics/claude-code @.> Cc: Michael Watson @.>; Comment @.> Subject: Re: [anthropics/claude-code] OpenTelemetry Monitoring: No Telemetry Data Exported (Issue #1712)

[https://avatars.githubusercontent.com/u/171466767?s=20&v=4]dzungtr left a comment (anthropics/claude-code#1712)https://github.com/anthropics/claude-code/issues/1712#issuecomment-2952482820

I was able to get it working now. The issue lie in the problem with my config for claude in $HOME/.claude/settings.json.

Instead of

"OTEL_EXPORTER_OTLP_PROTOCOL": "http",

It should be

"OTEL_EXPORTER_OTLP_PROTOCOL": "http/json", # or "http/protobuf"

Beside, there was problem with my observability set up. Otel-collector did not push data to Prometheus properly. Both of the silent errors and very hard to tell which part is wrong. Whoever come to this thread later, I recommend using some well packages observability stack like signoz. It work out of the box. It took me a day to try export with different method and console to debug the problem. Happy spending for AI :)

— Reply to this email directly, view it on GitHubhttps://github.com/anthropics/claude-code/issues/1712#issuecomment-2952482820, or unsubscribehttps://github.com/notifications/unsubscribe-auth/A6QOOYKA7HOP4DP2GKK75F33CLQ27AVCNFSM6AAAAAB6YI2EUOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDSNJSGQ4DEOBSGA. You are receiving this because you commented.

Michael Watson

@.@.>

hedgineer.iohttps://www.hedgineer.io

H E D G I N E E R

michael-hedgineer avatar Jun 07 '25 13:06 michael-hedgineer

Glad you figured it out!

sid374 avatar Jun 09 '25 07:06 sid374

i have otel working with prometheus, but the actual traces/spans are empty:

Trace ID: 
Span ID: 
Flags: 0

is this expected? i'm surprised that there are no examples of tracing claude code...


EDIT: i think i just didn't have export OTEL_TRACES_EXPORTER=otlp...now i seem to be able to find the content.

either way - as general feedback for the claude code team - i'm sure there are thousands of people that would benefit from a basic tutorial on how to trace claude code. even the claude docs helper bot doesn't know how to do it. this page is helpful but basically just a giant list: https://docs.anthropic.com/en/docs/claude-code/monitoring-usage


EDIT #2 : after much futzing around, this is the missing piece (the actual claude API response!) => https://github.com/anthropics/claude-code/issues/2090

rawwerks avatar Jun 14 '25 16:06 rawwerks

This issue has been automatically locked since it was closed and has not had any activity for 7 days. If you're experiencing a similar issue, please file a new issue and reference this one if it's relevant.

github-actions[bot] avatar Aug 10 '25 14:08 github-actions[bot]