Using otelcol.exporter.oltphttp to send logs to Loki OTLP points fails due to 204 response

[loafoe]

What's wrong?

Attempting to push logs from otelcol.exporter.olpthttp directly to a Loki 3.1.0 OTLP endpoint seems to fail i.e. logs are not acklowledged as being sent even though I can see the logs appear in Loki. This seems to be caused by Loki responding with a HTTP 204 which is not recognized as success by the oltphttp component.

I found a possible otel related upstream issue which was recently fixed:

https://github.com/open-telemetry/opentelemetry-cpp/issues/2632 https://github.com/open-telemetry/opentelemetry-cpp/pull/2712

If the otlphttp component is using this library then this would explain the above observation.

When I introduce a Caddy proxy in between with the following config, the problem goes away!

reverse_proxy /v1/logs {
    to loki-gateway.loki-system.svc:80
    @204 {
       status 204
    }
    handle_response @204 {
       copy_response 200
    }
    rewrite /otlp/v1/logs
}

Can alloy update to the latest opentelementry-cpp version to solve this?

Steps to reproduce

• Set up Alloy to push OTLP logs directly to a Loki 3.1.0 OTLP endpoint
• Observe the pipeline clogging up as delivery of logs is not acknowledged

System information

Linux grafana-alloy-1 6.1.92

Software version

Grafana Alloy v1.2.1

Configuration

loki.source.kubernetes "pods" {
  targets    = discovery.relabel.loki_relabel.output
  forward_to = [
    otelcol.receiver.loki.local.receiver,
  ]
}

otelcol.exporter.otlphttp "local" {
  client {
    # This endpoint rewrites /v1/logs to /otlp/v1/logs
    endpoint = "http://otlp-proxy.starlift-observability.svc"
    auth = otelcol.auth.headers.local.handler

    tls {
      insecure = false
    }
  }
}

Logs

ts=2024-07-12T08:18:33.463303168Z level=error msg="failed to consume log entries" component_path=/ component_id=otelcol.receiver.loki.local err="sending queue is full"

[github-actions[bot]]

This issue has not had any activity in the past 30 days, so the needs-attention label has been added to it. If the opened issue is a bug, check to see if a newer release fixed your issue. If it is no longer relevant, please feel free to close this issue. The needs-attention label signals to maintainers that something has fallen through the cracks. No action is needed by you; your issue will be kept open and you do not have to respond to this comment. The label will be removed the next time this job runs if there is new activity. Thank you for your contributions!

[algo7]

Any news on this?

[ebuildy]

it seems alloy dont take in account if the output stage is broken (queue full) or not, the input stage continue to read file but output stage drop data.

[carlosjgp]

Seeing this error too, using the latest Alloy (1.1.1) and Loki (6.30.1) charts

[subhajit-stylopay]

Experienced this issue too (Alloy v1.10.0) on the Alloy running on agent/ application server that writes to a remote loki