fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

plugin_proxy: enable event_type specification for proxy plugins

Open jmccormick7 opened this issue 2 months ago • 7 comments

This commit adds event_type to the flb_plugin_proxy_def struct. This allows for the setting of event_type to a value other than the default log only initialization.

In the flb_plugin_proxy_register function default behavior is enforced by checking for unset event_type values. When unset (a value of 0) then log behavior is defaulted. No need for incorrect value checking since this was not a set field in the past. Unset fields are set as current behavior dictates and future use cases using incorrect values can be treated as user error.

Input use case does not use event-type and is unaffected by this change.

Addresses #10995

This corresponding PR in the fluent-bit-go library enables the ability to allow Go plugins to register their event types PR 80


Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [N/A] Example configuration file for the change
  • [N/A] Debug log output from testing the change
  • [x] Attached Valgrind output that shows no leaks or memory corruption was found
100% tests passed, 0 tests failed out of 164

Label Time Summary:
internal    =  85.38 sec*proc (70 tests)
runtime     = 2026.55 sec*proc (83 tests)

Total Test time (real) = 2212.60 sec
==46444== 
==46444== HEAP SUMMARY:
==46444==     in use at exit: 0 bytes in 0 blocks
==46444==   total heap usage: 5,012 allocs, 5,012 frees, 54,842,164 bytes allocated
==46444== 
==46444== All heap blocks were freed -- no leaks are possible
==46444== 
==46444== For lists of detected and suppressed errors, rerun with: -s
==46444== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [ ] Run local packaging test showing all targets (including any new ones) build.
  • [ ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [ ] Documentation required for this feature

Backporting

  • [ ] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

Release Notes

  • New Features

    • Added event type support to proxy plugin definitions for enhanced plugin compatibility.
  • Tests

    • Added Go-based proxy plugin testing framework.
    • Introduced proxy logs compatibility test suite.
    • Expanded test environment configuration for Linux systems.
  • Chores

    • Updated proxy plugin registration logic.
    • Enhanced build configuration for test plugins.

✏️ Tip: You can customize this high-level summary in your review settings.

jmccormick7 avatar Oct 10 '25 18:10 jmccormick7

Walkthrough

This changeset introduces proxy plugin support for logs in Fluent Bit. It adds an event_type field to the plugin proxy definition structure, updates proxy registration logic to propagate event types, and implements comprehensive test infrastructure including a Go-based output plugin, test configuration, build scripts, and runtime shell tests.

Changes

Cohort / File(s) Summary
Plugin Proxy API
include/fluent-bit/flb_plugin_proxy.h
Added int event_type field to struct flb_plugin_proxy_def to store event type (logs/metrics/traces).
Plugin Proxy Implementation
src/flb_plugin_proxy.c
Modified flb_proxy_register_output to set output event type based on def->event_type (defaults to FLB_OUTPUT_LOGS if unset); added input plugin list registration in flb_proxy_register_input; changed flb_plugin_proxy_create to use flb_calloc instead of flb_malloc for proxy definition allocation.
Test Framework Configuration
tests/runtime_shell/CMakeLists.txt
Added Linux-specific conditional to include proxy_logs_expect.sh in shell unit tests; extended test environment to set FLB_BUILD variable alongside existing FLB_BIN.
Test Configuration & Build
tests/runtime_shell/conf/proxy_logs_test.conf, tests/runtime_shell/go_plugins/build_test_plugins.sh, tests/runtime_shell/go_plugins/go.mod
New test configuration defining Fluent Bit service, dummy input, and test output plugins; new build script ensuring Go installation, verifying CGO support, and compiling Go test plugin; new Go module declaring dependencies for fluent-bit-go and codec libraries.
Go Plugin Implementation
tests/runtime_shell/go_plugins/logs_output.go
New Go output plugin implementing logs-only support with standard lifecycle hooks (FLBPluginRegister, FLBPluginInit, FLBPluginFlushCtx, FLBPluginExit); decodes incoming records and prints them to stdout.
Runtime Test Script
tests/runtime_shell/proxy_logs_expect.sh
New shell test script executing Fluent Bit with proxy plugins, capturing output, and validating success/failure with stdout logging.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

  • Plugin proxy header and source modifications are straightforward but require verification that event type handling logic is correct across registration flows
  • New Go plugin follows standard Fluent Bit interface patterns but implementation should be checked for correct data decoding and output handling
  • Build script contains conditional logic and fallbacks that may have cross-platform implications
  • Test configuration and shell script dependencies need validation against actual test environment setup

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 33.33% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: adding event_type specification capability to proxy plugins, which is the primary focus of the changeset across all modified files.
✨ Finishing touches
🧪 Generate unit tests (beta)
  • [ ] Create PR with unit tests
  • [ ] Post copyable unit tests in a comment

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Oct 10 '25 18:10 coderabbitai[bot]

If we proceed to address this issue, we need to send another PR into here https://github.com/chronosphereio/calyptia-plugin to handle event_type as well.

cosmo0920 avatar Oct 17 '25 06:10 cosmo0920

If we proceed to address this issue, we need to send another PR into here https://github.com/chronosphereio/calyptia-plugin to handle event_type as well.

@cosmo0920 I can open a PR in that repository tonight!

jmccormick7 avatar Oct 17 '25 12:10 jmccormick7

@cosmo0920 How would you prefer I do testing beyond the unit tests? I've done some manual testing that I can document, but is there any formalized regression testing or tests for proxy plugins?

jmccormick7 avatar Nov 05 '25 15:11 jmccormick7

How about adding runtime shell test cases with precompiled Golang shared objects here?

https://github.com/fluent/fluent-bit/tree/master/tests/runtime_shell Or, leaving as-is is still good but it tends to be delayed for reviewing without unit test addtions.

cosmo0920 avatar Nov 07 '25 03:11 cosmo0920

@cosmo0920 I have added a runtime test that build and test a go plugin shared object. Please let me know if there are any improvements or other additions I can provide to speed up the timeline!

jmccormick7 avatar Nov 12 '25 17:11 jmccormick7

Plus, this PR still uses outdated base branch. So, could you rebase off the current master? No left device errors should be gone after rebasing off.

cosmo0920 avatar Nov 21 '25 08:11 cosmo0920