fluent
internal: add conditionals evaluation API
Description of the change
Implement the base API for conditionals evaluation using record accessor, this API will be initially used by a specific log processor, but eventually this same interface can be used as a generic conditional evaluation for processor.
This PR:
- Implements the supported conditional operators (AND/OR)
- Implements the supported rule operators (Listed below)
- Types
- Library public interface
- Unit testing covering all conditional and rule operators
Further description
Supported condition operators
The following are the operator types supported for conditions
enum condition_operator {
COND_OP_AND,
COND_OP_OR
};
Supported rule operators
The following are the operator types supported for each rule
enum flb_rule_operator {
FLB_RULE_OP_EQ,
FLB_RULE_OP_NEQ,
FLB_RULE_OP_GT,
FLB_RULE_OP_LT,
FLB_RULE_OP_REGEX,
FLB_RULE_OP_IN,
FLB_RULE_OP_NOT_IN
};
Types
The following are the basic types to support both conditional and rules.
struct flb_condition_rule {
struct flb_cfl_record_accessor *ra; /* Record accessor for the field */
enum record_context_type context; /* Whether rule applies to body or metadata */
enum flb_rule_operator op;
union {
flb_sds_t str_val;
double num_val;
struct {
flb_sds_t *values;
int count;
} array;
} value;
struct flb_regex *regex;
struct mk_list _head;
};
struct flb_condition {
enum flb_condition_operator op;
struct mk_list rules;
};
Library interface
The following is the proposed interface for parsing conditionals and evaluating rules.
struct flb_condition *flb_condition_create(enum flb_condition_operator op);
int flb_condition_add_rule(struct flb_condition *cond,
const char *field,
enum flb_rule_operator op,
void *value,
int value_count,
enum record_context_type context);
void flb_condition_destroy(struct flb_condition *cond);
int flb_condition_evaluate(struct flb_condition *cond,
struct flb_mp_chunk_record *record);
Tests
Tests are passing and no address sanitizer issue has been found
Test equals... [ OK ]
Test not_equals... [ OK ]
Test numeric... [ OK ]
Test numeric_edge_cases... [ OK ]
Test in... [ OK ]
Test not_in... [ OK ]
Test regex... [ OK ]
Test and... [ OK ]
Test or... [ OK ]
Test empty... [ OK ]
Test invalid_expressions... [2024/12/19 09:31:40] [error] [record accessor] syntax error, unexpected '[', expecting IDENTIFIER at '$[invalid'
[ OK ]
Test metadata... [ OK ]
Test missing_values... [ OK ]
Test border_cases... [ OK ]
SUCCESS: All unit tests have passed.
Usage of the API
/* Test both conditions true */
record_data = create_test_record("level", "error");
TEST_CHECK(record_data != NULL);
cond = flb_condition_create(FLB_COND_OP_AND);
TEST_CHECK(cond != NULL);
TEST_CHECK(flb_condition_add_rule(cond, "$level", FLB_RULE_OP_EQ,
"error", 0, RECORD_CONTEXT_BODY) == FLB_TRUE);
TEST_CHECK(flb_condition_add_rule(cond, "$level", FLB_RULE_OP_NEQ,
"info", 0, RECORD_CONTEXT_BODY) == FLB_TRUE);
result = flb_condition_evaluate(cond, &record_data->chunk);
TEST_CHECK(result == FLB_TRUE);
How this will be used
- type: filter
condition:
operator: AND
rules:
- field: "$kubernetes.namespace"
operator: eq
value: "production"
- field: "$log.level"
operator: in
value: ["error", "fatal"]
- field: "$request.latency"
operator: gt
value: 1000
- field: "$request.path"
operator: regex
value: "^/api/v[0-9]/users/.*$"
- field: "$kubernetes.labels.team"
operator: eq
context: metadata
value: "platform"
- type: block_keys
opts:
regex: "credential|token"
condition:
operator: OR
rules:
- field: "$metadata.env"
operator: eq
value: "prod"
- field: "$security.classification"
operator: in
value: ["restricted", "confidential"]
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Leaksanitizer complains that there are memory leaks. https://github.com/fluent/fluent-bit/actions/runs/12409308281/job/34642662195?pr=9749#step:5:3675 Could you check them?
Leaksanitizer complains that there are memory leaks. https://github.com/fluent/fluent-bit/actions/runs/12409308281/job/34642662195?pr=9749#step:5:3675 Could you check them?
Done