TLS : OpenSSL Provider and Store API support to support HSMs

[Brent-Crawley-GGL]

Added support for loading OpenSSL Providers. Up to 4 Providers that are defined in the OpenSSL configuration file can be activated by specifying the name in the new configuration option. This isn't specific to TLS, and can be used in general to load Providers for specific functionality.

Added a new compile time option to support the OpenSSL Store API (FLB_OPENSSL_STORE). This will enable treating the TLS crt_file and key_file options as Store URIs. There is also the ability to define a Provider Query to further help with utilising Providers in this scenario. With this ability it becomes possible to load the TLS artifacts from an HSM.

---

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

• [✅] Example configuration file for the change

service:
  log_level: debug
  openssl.providers: c7000

pipeline:
  inputs:
    - name: dummy
      dummy: '{"message": "Hello Joe"}'
 
  outputs:
    - name: opentelemetry
      match: '*'
      host: '192.168.8.1'
      port: 8909
      metrics_uri: /v1/metrics
      logs_uri: /v1/logs
      traces_uri: /v1/traces
      log_response_payload: true
      tls: on
      tls.verify: off
      tls.crt_file: 'GGL-STORE://type=cert;alias=dsscert'
      tls.key_file: 'GGL-STORE://type=priv;alias=dsskeyecc'
      tls.provider_query: '-fips, ?provider=gallagher' 

• [✅] Debug log output from testing the change

Fluent Bit v4.0.2
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io
 
______ _                  _    ______ _ _             ___  _____
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/
 
 
[2025/07/18 10:33:27] [ info] Configuration:
[2025/07/18 10:33:27] [ info]  flush time     | 1.000000 seconds
[2025/07/18 10:33:27] [ info]  grace          | 5 seconds
[2025/07/18 10:33:27] [ info]  daemon         | 0
[2025/07/18 10:33:27] [ info] ___________
[2025/07/18 10:33:27] [ info]  inputs:
[2025/07/18 10:33:27] [ info]      tail
[2025/07/18 10:33:27] [ info]      tail
[2025/07/18 10:33:27] [ info]      tail
[2025/07/18 10:33:27] [ info] ___________
[2025/07/18 10:33:27] [ info]  filters:
[2025/07/18 10:33:27] [ info] ___________
[2025/07/18 10:33:27] [ info]  outputs:
[2025/07/18 10:33:27] [ info]      opentelemetry.0
[2025/07/18 10:33:27] [ info] ___________
[2025/07/18 10:33:27] [ info]  collectors:
[2025/07/18 10:33:27] [debug] [tls] init
[2025/07/18 10:33:27] [debug] [tls] resetting providers
[2025/07/18 10:33:27] [debug] [tls] loading provider 'c7000'
[2025/07/18 10:33:28] [ info] [fluent bit] version=4.0.2, commit=47cf86504b, pid=592
[2025/07/18 10:33:28] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2025/07/18 10:33:28] [ info] [storage] ver=1.5.3, type=memory+filesystem, sync=full, checksum=off, max_chunks_up=128
[2025/07/18 10:33:28] [ info] [storage] backlog input plugin: storage_backlog.3
[2025/07/18 10:33:28] [ info] [simd    ] disabled
[2025/07/18 10:33:28] [ info] [cmetrics] version=1.0.2
[2025/07/18 10:33:28] [ info] [ctraces ] version=0.6.6
[2025/07/18 10:33:28] [ info] [input:tail:tail.0] initializing
[2025/07/18 10:33:28] [ info] [input:tail:tail.0] storage_strategy='filesystem' (memory + filesystem)
...
[2025/07/18 10:33:28] [debug] [tls] Loading store item; query: -fips, ?provider=gallagher, uri: GGL-STORE://type=cert;alias=dss                                                                                                              cert, arg: (nil)
[2025/07/18 10:33:28] [debug] [tls] Loading store item; query: -fips, ?provider=gallagher, uri: GGL-STORE://type=priv;alias=dss 

This was tested on an embedded device, running an i.mx8 Processor with an EdgeLock chip for Secure World access.

• [❌] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

• [❌] Run local packaging test showing all targets (including any new ones) build.
• [❌] Set ok-package-test label to test for all targets (requires maintainer to do).

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

• New Features

  • OpenSSL 3 provider and store support for TLS, including loading certificates/keys from OpenSSL Store URIs.
  • Global TLS provider management with initialization and cleanup.

• Configuration

  • New service option: openssl.providers.
  • New TLS option: tls.provider_query (available for inputs, outputs, and upstream nodes).
  • Build-time option to enable OpenSSL Store support.

• Plugins

  • Added tls.provider_query support across major outputs (e.g., CloudWatch, Kinesis, S3, OpenSearch/ES, BigQuery) and relevant filters/OAuth flows.

• Chores

  • Updated .gitignore to exclude .vs and build outputs.

[patrick-stephens]

Sounds a good idea, one thing though is we'll definitely need some tests for it and docs - particularly for non-Linux platforms as that is often a nightmare to figure out.

At the moment this is disabled for everything, I would like to see us prove it builds for all targets (that support it) by enabling it appropriately and adding any missing dependencies (presumably none).

[coderabbitai[bot]]

Walkthrough

Introduces OpenSSL provider/store support, adds TLS provider_query configuration and propagation, extends TLS APIs and backend signatures, updates plugins/tests to new TLS create signature, and wires global TLS configure/cleanup into runtime. Adds CMake option FLB_OPENSSL_STORE and ignores .vs/out in .gitignore.

Changes

Cohort / File(s)	Summary
Build and Ignore Config CMakeLists.txt, .gitignore	Adds FLB_OPENSSL_STORE option with FLB_TLS dependency and macro define; ignores .vs and /out/.
Public Config and Structs include/fluent-bit/flb_config.h	Adds openssl_providers field and FLB_CONF_STR_OPENSSL_PROVIDERS key.
Input/Output/Public TLS Fields include/fluent-bit/flb_input.h, include/fluent-bit/flb_output.h, include/fluent-bit/flb_upstream_node.h	Adds tls_provider_query to input/output/upstream node structs; updates flb_upstream_node_create to accept tls_provider_query.
TLS Public API include/fluent-bit/tls/flb_tls.h	Extends flb_tls_create and backend context_create with extra const char*; adds flb_tls_configure and flb_tls_cleanup declarations.
Core Config Wiring src/flb_config.c	Parses/initializes/frees openssl_providers; adds service config entry.
Core Input/Output Wiring src/flb_input.c, src/flb_output.c	Adds tls.provider_query parsing, storage, cleanup; passes to flb_tls_create.
Runtime Env Hooks src/flb_lib.c	Adds flb_configure_env calling flb_tls_configure; calls flb_tls_cleanup on stop; makes flb_init_env(void).
Upstream HA/Node src/flb_upstream_ha.c, src/flb_upstream_node.c	Adds tls.provider_query handling; extends node creation; passes to flb_tls_create; manages memory.
TLS Implementation src/tls/flb_tls.c, src/tls/openssl.c	Adds tls.provider_query config map; extends flb_tls_create and backend pass-through; implements OpenSSL 3 provider loading/store support, configure/cleanup, and provider-based cert/key loading.
Plugins updated to new TLS signature (provider_query) plugins/out_bigquery/bigquery.c, plugins/out_cloudwatch_logs/cloudwatch_logs.c, plugins/out_es/es_conf.c, plugins/out_kinesis_firehose/firehose.c, plugins/out_kinesis_streams/kinesis.c, plugins/out_opensearch/os_conf.c, plugins/out_s3/s3.c, src/aws/flb_aws_credentials.c	Appends ins->tls_provider_query to flb_tls_create calls (multiple sites each).
Plugins updated with NULL placeholder plugins/filter_kubernetes/kube_meta.c, plugins/filter_nightfall/nightfall.c, plugins/in_kubernetes_events/kubernetes_events_conf.c, plugins/out_azure_blob/azure_blob_conf.c, src/flb_oauth2.c	Adds extra trailing NULL to flb_tls_create.
Azure Kusto Upstream Node plugins/out_azure_kusto/azure_kusto_conf.c	Adds extra NULL to flb_upstream_node_create per new parameter.
Tests tests/runtime/in_tcp.c, tests/runtime/out_tcp.c	Update flb_tls_create calls with additional NULL argument.

Sequence Diagram(s)

sequenceDiagram
  autonumber
  actor App
  participant Lib as flb_lib
  participant TLS as flb_tls.c
  participant OSSL as openssl.c

  App->>Lib: do_start()
  Lib->>TLS: flb_tls_configure(config)
  TLS->>OSSL: tls_configure(config.openssl_providers)
  OSSL-->>TLS: Providers loaded (if OpenSSL 3)
  TLS-->>Lib: configured

  App->>TLS: flb_tls_create(..., key_passwd, provider_query)
  TLS->>OSSL: tls_context_create(..., provider_query)
  alt provider_query/URI supported
    OSSL->>OSSL: Load cert/key from provider/store
  else fallback
    OSSL->>OSSL: Load cert/key from files
  end
  OSSL-->>TLS: ctx*
  TLS-->>App: flb_tls*

  App-->>TLS: flb_tls_cleanup() on shutdown
  TLS->>OSSL: tls_cleanup()
  OSSL-->>TLS: providers reset

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Suggested reviewers

• leonardo-albertovich
• koleini
• patrick-stephens
• fujimotos
• edsiper

Poem

A rabbit twitches whiskers, keen and sure,
New TLS paths through OpenSSL’s burrowed door.
Providers hop in, stores unlock the key,
Queries nibble secrets from a leafy tree.
With tidy NULLs and configs set to gleam—
We bound toward handshakes, fast as a dream. 🐇🔐

[!TIP]

🔌 Remote MCP (Model Context Protocol) integration is now available!

Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats.

✨ Finishing Touches

• [ ] 📝 Generate Docstrings

🧪 Generate unit tests

• [ ] Create PR with unit tests
• [ ] Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

🪧 Tips

Chat

There are 3 ways to chat with CodeRabbit:

• Review comments: Directly reply to a review comment made by CodeRabbit. Example:
  • I pushed a fix in commit <commit_id>, please review it.
  • Open a follow-up GitHub issue for this discussion.
• Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query.
• PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
  • @coderabbitai gather interesting stats about this repository and render them as a table. Additionally, render a pie chart showing the language distribution in the codebase.
  • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.

Support

Need help? Create a ticket on our support page for assistance with any issues or questions.

CodeRabbit Commands (Invoked using PR/Issue comments)

Type @coderabbitai help to get the list of available commands.

Other keywords and placeholders

• Add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.
• Add @coderabbitai summary to generate the high-level summary at a specific location in the PR description.
• Add @coderabbitai anywhere in the PR title to generate the title automatically.

CodeRabbit Configuration File (.coderabbit.yaml)

• You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
• Please see the configuration documentation for more information.
• If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Status, Documentation and Community

• Visit our Status Page to check the current availability of CodeRabbit.
• Visit our Documentation for detailed information on how to use CodeRabbit.
• Join our Discord Community to get help, request features, and share feedback.
• Follow us on X/Twitter for updates and announcements.

[github-actions[bot]]

This PR is stale because it has been open 45 days with no activity. Remove stale label or comment or this will be closed in 10 days.