fluent-bit
fluent-bit copied to clipboard
opentelemetry output not sending trace_id and span_id for logs
trafficstars
Bug Report
Describe the bug opentelemetry output does not send correctly trace_id and span_id
To Reproduce
[SERVICE]
Flush 1
Daemon Off
Log_Level info
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 8888
storage.path /tmp/storage
storage.sync normal
storage.checksum off
storage.backlog.mem_limit 5M
# Parsers_File /fluent-bit/etc/parsers.conf
# OpenTelemetry Input for gRPC
[INPUT]
Name opentelemetry
Listen 0.0.0.0
Port 4317
Buffer_Max_Size 10MB
Buffer_Chunk_Size 1MB
Log_Level debug
# Debug filter to print all records
[FILTER]
Name stdout
Match *
# Route logs to Loki using OTLP over HTTP
[OUTPUT]
Name opentelemetry
Match v1_logs
Host loki
Port 3100
Logs_uri /otlp/v1/logs
grpc false
http2 false
Log_response_payload true
Log_Level trace
Workers 4
tls Off
tls.verify Off
logs_trace_id_metadata_key trace_id
logs_span_id_metadata_key span_id
when ingesting logs here an example input :
2025-06-05T15:52:05.646989193Z [0] v1_logs: [1749138711.18446744072203624508, {"otlp"=>{"observed_timestamp"=>1749138711578345763, "timestamp"=>1749138711578090835, "severity_number"=>9, "severity_text"=>"INFO", "trace_id"=>"1\"\x0aK\xad\xd9\x11)\xb6\xda\xcfg\x1dp\xefE", "span_id"=>"h\x05\xde0|(\x04\xea", "trace_flags"=>1}}, {"message"=>"API Gateway received provision request for client: unknown, traceId: 31220a4badd91129b6dacf671d70ef45"}]
as you could see the trace_id and span_id have been ingested correctly by the opentelemetry input but as you will see later all resources metadata and structured metadata have been ingested correctly except for the trace_id and span_id
Expected behavior
trace_id and span_id should be sent correctly to loki using opentelemetry output for logs. I tested for metrics and traces and Prometheus and Tempo were able to ingest them correctly and to have trace_id and span_id without any difficulty.
Screenshots
Your Environment
- Version used: 4.0.3
- Configuration:
- Environment name and version (e.g. Kubernetes? What version?): docker compose
- Server type and version: local
- Operating System and version: Windows 10
- Filters and plugins:
Additional context
- i'm trying to create an observability data plateform where i'm having data pipeline from Spring Boot applications to Fluent Bit as OpenTelemetry collector which later forward to loki, tempo and prometheus in otlp format.
- Everything is working correctly except for the trace_id and span_id for the logs.