OpenTelemetry - Metrics Endpoint Giving 404 Not Found

[nayakmk]

trafficstars

Describe the bug

I am using open-collector Kubernetes to collect the application traces, metrics, logs via OTLP.

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yml"]
    volumes:
      - ./otel-collector-config.yml:/etc/otel-collector-config.yml
    ports:
      - "1888:1888"   # pprof extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver
      - "55679:55679" # zpages extension

I can send the traces via localhost:4318/v1/traces but for the metrics ( localhost:4318/v1/metrics ) when the API invocation happens it gives 404.

Steps to reproduce

Create a spring boot project with starter dependency

implementation 'io.opentelemetry.instrumentation:opentelemetry-spring-boot-starter'

Add VM Argument:

-Dserver.port=8089 -DOTEL_SERVICE_NAME=search_service_java -DOTEL_TRACES_EXPORTER=otlp -DOTEL_METRICS_EXPORTER=otlp -DOTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318

What did you expect to see? The metrics endpoint should be present to receive metrics data.

What did you see instead? Got 404 error

What version did you use?

Docker Image: open-collector - latest opentelemetry-spring-boot-starter - 1.34.1 io.opentelemetry.instrumentation:opentelemetry-instrumentation-bom-alpha:2.0.0-alpha

What config did you use?

   receivers:
     otlp:
       protocols:
         grpc:
         http:
   exporters:
     otlphttp/prometheus:
       endpoint: http://prometheus:9090/api/v1/otlp
       tls:
         insecure: true
     otlp/jaeger:
       endpoint: jaeger-all-in-one:14250
       tls:
         insecure: true 
     logging:
   processors:
     batch:
   extensions:
     health_check:
   service:
     telemetry:
       logs:
         level: "debug"
     extensions: [health_check]
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [logging, otlp/jaeger]
       metrics: 
         receivers: [otlp]
         processors: [batch]
         exporters: [otlphttp/prometheus]

Environment Windows

Additional context

[nayakmk]

Any update to this ?

[codeboten]

It's hard to gauge there the problem may lie here since the setup has a spring app, a collector, and a kubernetes deployment. Can you confirming that pinging the endpoint directly returns a 404? Based on your configuration for the collector, it looks like that endpoint should be responsive, maybe the config isn't taking effect? That would be another area I would investigate

[adoublef]

I am currently getting a similar error. using this Go application I can't seem to tell why the /v1/metrics endpoint can't be found. The /v1/traces does seem to exist so I am wondering if there is something that needs to be set in the collector config to enable metrics to work.

[christiangda]

Hi Everyone,

I had the same issue and I found the fix by reading this thread https://stackoverflow.com/questions/79085645/unable-to-write-otlp-data-to-the-prometheus-otlp-write-receiver.

This was a breaking change from 2.x.y --> 3.x.y

My solution was to change my Prometheus configuration: FROM: --enable-feature=otlp-write-receiver --> --web.enable-otlp-receiver

TO:

    - name: prometheus
      image: docker.io/prom/prometheus:latest
      args:
        - "--config.file=/etc/prometheus/prometheus.yml"
        - "--storage.tsdb.path=/prometheus"
        - "--web.console.libraries=/etc/prometheus/console_libraries"
        - "--web.console.templates=/etc/prometheus/consoles"
        - "--web.enable-lifecycle"
        - "--web.enable-otlp-receiver"

If you want to check my go template project here you are: https://github.com/p2p-b2b/go-rest-api-service-template

and my configuration: Prometheus configuration