fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Lua filter before opentelemetry_envelope does not work

Open count0ru opened this issue 6 months ago • 1 comments

Bug Report

Describe the bug I'm collecting logs in syslog format, then adding custom fields to opentelemetry attributes before sending them to opentelemetry. I need to extract the host field from the syslog message and map it to the opentelemetry attribute host.name. To achieve this, I'm using a Lua script to enrich messages for OpenTelemetry, (otel_resource_attributes only supports text values, as I understand it)

To Reproduce

  • place Lua filter before 'opentelemetry_envelope'
[0] bmclogs: [1748878869.000000000, {}, {"pri"=>"174", "time"=>"Jun  2 15:41:09", "host"=>"myhost", "ident"=>"maint_debug_mod", "message"=>"2025-06-02 15:41:09 Redfish,user@myhost,redfish,User(user) authentication failed"}]
[2025/06/02 15:41:15] [debug] [processor] no condition set for processor unit (pu=opentelemetry_envelope)
[0] bmclogs: [1748878869.000000000, {}, {"pri"=>"174", "attributes"=>{"resource.id"=>"myresourceid", "service.name"=>"mycollector", "product.code"=>"myproduct", "host.name"=>"myhost"}, "time"=>"Jun  2 15:41:09", "ident"=>"maint_debug_mod", "host"=>"myhost", "message"=>"2025-06-02 15:41:09 Redfish,user@myhost,redfish,User(user) authentication failed"}]

Expected behavior The Lua script should add the required fields, allowing me to reference them via logs_attributes_metadata_key.

Your Environment

  • Version used: Fluent Bit v4.0.1
  • Operating System and version: Ubuntu 22.04.5 LTS
  • Configuration:
pipeline:
  inputs:

    - name: syslog
      tag: bmclogs
      listen: ::0
      parser: syslog-rfc3164
      mode: tcp
      port: 5140
      processors:
        logs:
          - name: stdout
          - name: lua
            code: |
              function cb_modify(tag, timestamp, record)
                  local service_name = "mycollector"
                  local product_code = "myproduct"
                  local host_name = record["host"] or "unknown"
                  local resource_id = "myresourceid"
                  local new_record = record
                  new_record["attributes"] = {
                      ["resource.id"] = resource_id,
                      ["service.name"] = service_name,
                      ["product.code"] = product_code,
                      ["host.name"] = host_name
                  }
                  return 1, timestamp, new_record
              end
            call: cb_modify

          - name: opentelemetry_envelope
  outputs:    
   - name: opentelemetry
      match: bmclogs
      host: collector.local
      port: 4318
      logs_uri: /v1/logs
      tls: true
      tls.verify: true
      log_response_payload: true
      logs_attributes_metadata_key: $attributes

count0ru avatar Jun 02 '25 15:06 count0ru

WIP: Lua filter enhancements: https://github.com/fluent/fluent-bit/pull/10457

edsiper avatar Jun 09 '25 23:06 edsiper

fixed in https://github.com/fluent/fluent-bit/pull/10444

edsiper avatar Jul 08 '25 17:07 edsiper