unexpected event 'mapping-start-event' (9) in state 'section-value' (5)

[ramdaspotale]

Bug Report

Describe the bug I am trying to use yaml configuration for fluent bit config because you need yaml config in order to use content modifier filter as per its docs here: https://docs.fluentbit.io/manual/pipeline/processors/content-modifier. so after converting my fluent bit config from classic to yaml, fluent bit is not starting and throwing error like below. i also could not find full yaml sample config file for fluent bit.

[2024/09/27 12:17:24] [error] [config] YAML error found in file "/fluent-bit/etc/conf/fluent-bit.yaml", line 11, column 4: unexpected event 'mapping-start-event' (9) in state 'section-value' (5). [2024/09/27 12:17:24] [error] yaml error [2024/09/27 12:17:24] [error] configuration file contains errors, aborting.

To Reproduce

• use yaml config with fluent bit on kubernetes environment and fluent bit daemonset pods do not start.

Expected behavior i expect fluent bit to start normally and start processing logs

Your Environment

• Version used: Fluent Bit v3.0.7
• Configuration:

fluent-bit.yaml:

SERVICE:
  Daemon: Off
  Flush: 1
  Log_Level: debug
  HTTP_Server: On
  Parsers_File: /fluent-bit/etc/conf/custom_parsers.yaml
  HTTP_Listen: 0.0.0.0
  HTTP_Port: 2020
  Health_Check: On
  storage:
    path: /fluent-bit/buffer-storage
    sync: full
    max_chunks_up: 256


INPUT:
- Name: tail
  Path: /var/log/containers/*.log
  multiline:
    parser: docker, cri
  Tag: kube.*
  Mem_Buf_Limit: 100MB
  Skip_Long_Lines: On
  storage:
    type: filesystem
  DB: /fluent-bit/buffer-storage/fluent-buffer.db
  Exclude_Path:
    - /var/log/containers/fluent-bit-*.log
    - /var/log/containers/heartbeat-*.log
    - /var/log/containers/metricbeat*.log
    - /var/log/containers/*calico-system*.log
    - /var/log/containers/microsoft-defender-*.log
    - /var/log/containers/*gatekeeper-system*.log
    - /var/log/containers/*kube-system*.log
    - /var/log/containers/*kube-public*.log
    - /var/log/containers/*kube-node-lease*.log
    - /var/log/containers/*tigera-operator*.log
    - /var/log/containers/*prometheus*.log
    - /var/log/containers/fluent-daemon*.log

- Name: systemd
  Tag: host.*
  Systemd_Filter: _SYSTEMD_UNIT=kubelet.service
  Read_From_Tail: On
  storage:
    type: filesystem

- Name: forward
  Listen: 0.0.0.0
  Port: 2025
  Tag: idx.*
  Buffer_Chunk_Size: 100M
  Buffer_Max_Size: 500M

- Name: fluentbit_metrics
  Tag: internal_metrics
  Scrape_interval: 2

FILTER:
- Name: modify
  Match: idx.*
  Add:
    log_source: ics_idx

- Name: kubernetes
  Match: kube.*
  Merge_Log: On
  Keep_Log: Off
  Merge_Log_Key: event
  K8S-Logging:
    Parser: On
    Exclude: On
  Annotations: Off
  Buffer_Size: 400MB
  Labels: On
  Namespace_Labels: On

- Name: modify
  Match: kube.*
  Remove:
    - kubernetes.docker_id
    - kubernetes.container_name
    - kubernetes.pod_id
    - time
    - kubernetes.container_hash

- Name: nest
  Match: kube.*
  Operation: lift
  Nested_under: event
  Add_prefix: event_

- Name: nest
  Match: kube.*
  Operation: lift
  Nested_under: event_log
  Add_prefix: event_log_

- Name: modify
  Match: kube.*
  Rename:
    event_Level: level
    event_level: level
    event_log_Level: level
    event_message: message
    event_@timestamp: '@timestamp'
    log: message
    event_msg: message
  Remove:
  - event_ts

OUTPUT:
- Name: prometheus_remote_write
  Match: internal_metrics
  Host: ${CORTEX_HOST}
  Port: 443
  Uri: /api/v1/push
  Header:
    X-Scope-OrgID: ${CLUSTER_PREFIX}
  Log_response_payload: True
  Tls: On
  Tls.verify: Off
  add_label:
    - node: ${MY_POD_IP}
    - cluster: ${CLUSTER_PREFIX}
    - fluent_node_os: linux

- Name: es
  Match: '*'
  Host: ${FLUENT_ELASTICSEARCH_HOST}
  Port: 9200
  HTTP_User: ${FLUENT_ELASTICSEARCH_USER}
  HTTP_Passwd: ${FLUENT_ELASTICSEARCH_PASSWORD}
  Logstash_Format: On
  Logstash_Prefix: ${FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX}
  Time_Key: '@timestamp'
  Trace_Error: On
  Trace_Output: On
  tls: On
  tls.verify: Off
  Suppress_Type_Name: On
  Pipeline: ${FLUENT_ELASTICSEARCH_INGEST_PIPELINE}
  Retry_Limit: False
  Buffer_Size: 400MB
  Replace_Dots: On
  storage:
    total_limit_size: 500MB

• Environment name and version: Kubernetes (AKS) v1.30.4 and v1.28.5
• Server type and version: AKS
• Filters and plugins: all included in config above

Additional context I am trying to use content modifier to rename and drop few fields in log which filter is unable to do it. but as fluent bit is not starting with yaml config i am stuck.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.