fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

opentelemetry output not sending trace_id and span_id for logs

Open raedkit opened this issue 5 months ago • 5 comments
trafficstars

Bug Report

Describe the bug opentelemetry output does not send correctly trace_id and span_id

To Reproduce

[SERVICE]
    Flush        1
    Daemon       Off
    Log_Level    info
    HTTP_Server  On
    HTTP_Listen  0.0.0.0
    HTTP_Port    8888
    storage.path  /tmp/storage
    storage.sync  normal
    storage.checksum  off
    storage.backlog.mem_limit  5M
#    Parsers_File /fluent-bit/etc/parsers.conf

# OpenTelemetry Input for gRPC
[INPUT]
    Name        opentelemetry
    Listen      0.0.0.0
    Port        4317
    Buffer_Max_Size  10MB
    Buffer_Chunk_Size  1MB
    Log_Level   debug

# Debug filter to print all records
[FILTER]
    Name    stdout
    Match   *

# Route logs to Loki using OTLP over HTTP
[OUTPUT]
    Name        opentelemetry
    Match       v1_logs
    Host        loki
    Port        3100
    Logs_uri    /otlp/v1/logs
    grpc        false
    http2       false
    Log_response_payload true
    Log_Level   trace
    Workers     4
    tls         Off
    tls.verify  Off
    logs_trace_id_metadata_key   trace_id
    logs_span_id_metadata_key    span_id

when ingesting logs here an example input :

2025-06-05T15:52:05.646989193Z [0] v1_logs: [1749138711.18446744072203624508, {"otlp"=>{"observed_timestamp"=>1749138711578345763, "timestamp"=>1749138711578090835, "severity_number"=>9, "severity_text"=>"INFO", "trace_id"=>"1\"\x0aK\xad\xd9\x11)\xb6\xda\xcfg\x1dp\xefE", "span_id"=>"h\x05\xde0|(\x04\xea", "trace_flags"=>1}}, {"message"=>"API Gateway received provision request for client: unknown, traceId: 31220a4badd91129b6dacf671d70ef45"}]

as you could see the trace_id and span_id have been ingested correctly by the opentelemetry input but as you will see later all resources metadata and structured metadata have been ingested correctly except for the trace_id and span_id

Expected behavior trace_id and span_id should be sent correctly to loki using opentelemetry output for logs. I tested for metrics and traces and Prometheus and Tempo were able to ingest them correctly and to have trace_id and span_id without any difficulty.

Screenshots

Image

Your Environment

  • Version used: 4.0.3
  • Configuration:
  • Environment name and version (e.g. Kubernetes? What version?): docker compose
  • Server type and version: local
  • Operating System and version: Windows 10
  • Filters and plugins:

Additional context

  • i'm trying to create an observability data plateform where i'm having data pipeline from Spring Boot applications to Fluent Bit as OpenTelemetry collector which later forward to loki, tempo and prometheus in otlp format.
  • Everything is working correctly except for the trace_id and span_id for the logs.

raedkit avatar Jun 11 '25 14:06 raedkit