out_es: custom http headers

[AndrewChubatiuk]

Added ability to set custom HTTP headers for Elasticsearch output

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [x] Example configuration file for the change

[INPUT]
    name              tail
    path              /var/lib/docker/containers/**/*.log
    path_key         path
    multiline.parser  docker, cri
    Parser docker
    Docker_Mode  On

[INPUT]
    Name     syslog
    Listen   0.0.0.0
    Port     5140
    Parser   syslog-rfc3164
    Mode     tcp

[SERVICE]
    Flush        1
    Parsers_File parsers.conf

[Output]
    Name es
    Match *
    host victorialogs
    port 9428
    compress gzip
    path /insert/elasticsearch
    header AccountID 0
    header ProjectID 0
    header VL-Stream-Fields path
    header VL-Msg-Field log
    header VL-Time-Field @timestamp

• [N/A] Debug log output from testing the change

• [N/A] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [N/A] Run local packaging test showing all targets (including any new ones) build.
• [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

• [x] Documentation required for this feature

https://github.com/fluent/fluent-bit-docs/pull/1390

Backporting

• [x] Backport to latest stable release.

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

[nunopintocicd]

@PettitWesley @edsiper any news on this ?

[AndrewChubatiuk]

hey @edsiper could this feature be also included in release 3.2?

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[strawgate]

Is there anything we can do to get this included in a future release?

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.

[coderabbitai[bot]]

Walkthrough

Adds support for user-defined HTTP headers in the Elasticsearch output plugin. Introduces a headers field in the context, config map entry for multiple header pairs, and iterates configured headers to inject them into HTTP requests during flush and initial request setup.

Changes

Cohort / File(s)	Summary of Changes
Header configuration plumbing plugins/out_es/es.h, plugins/out_es/es.c	Adds ctx->headers field to struct flb_elasticsearch and exposes a new FLB_CONFIG_MAP_SLIST_1 config entry "header" (MULT true) to accept multiple header key/value pairs.
HTTP request header injection plugins/out_es/es.c	Iterates ctx->headers at flush/request setup, extracting key/value pairs and adding them to outgoing HTTP requests. Introduces traversal vars (head, mv, key, val).

Sequence Diagram(s)

sequenceDiagram
    autonumber
    participant FB as Fluent Bit
    participant ES as ES Output Plugin
    participant HC as HTTP Client
    participant ELS as Elasticsearch

    FB->>ES: flush(cb_es_flush)
    Note over ES: Load config (ctx), incl. headers list
    ES->>HC: build request (URI, payload)
    loop For each configured header
        ES->>HC: addHeader(key, value)
    end
    HC->>ELS: HTTP request with headers
    ELS-->>HC: HTTP response
    HC-->>ES: deliver response
    ES-->>FB: flush result

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

Poem

I tuck new keys in my satchel tight,
Hop-hop I go, by moonlit byte—
Headers stitched to every quest,
Whispered names on each request.
Elasticsearch hears my tap-tap-tap,
A rabbit courier with an HTTP cap.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title Check	✅ Passed	The title clearly and concisely summarizes the primary feature—adding custom HTTP headers support for the Elasticsearch output plugin—using the established “out_es” prefix and specific terminology without unnecessary detail or ambiguity.

✨ Finishing touches

• [ ] 📝 Generate Docstrings

🧪 Generate unit tests

• [ ] Create PR with unit tests
• [ ] Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🧪 Early access (Sonnet 4.5): enabled

We are currently testing the Sonnet 4.5 model, which is expected to improve code review quality. However, this model may lead to increased noise levels in the review comments. Please disable the early access features if the noise level causes any inconvenience.

Note:

• Public repositories are always opted into early access features.
• You can enable or disable early access features from the CodeRabbit UI or by updating the CodeRabbit configuration file.

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[eschabell]

@patrick-stephens can you review this PR?

[patrick-stephens]

Looks fine to me although I was expecting us to just use the HTTP plugin configuration for this, plus we probably want to replicate it to Opensearch if not there already.