fluent-bit
fluent-bit copied to clipboard
High filesystem read activity after upgrade without corresponding throughput increase
After upgrading from Fluent Bit version 3.0.7 to 3.1.2, I noticed a significant increase in the container_fs_reads_bytes_total metric. With version 3.0.7, values typically ranged from tens to hundreds of kilobytes. After upgrading — with no changes in configuration — the minimum values are now in megabytes, and the average is in the tens of MB.
In our production environment, this triggers alerts under higher load, where many more containers are running. The difference in filesystem read activity between the versions is therefore very large and concerning.
I would like to ask: Which specific change or commit could have introduced this behavior? Is there a parameter or setting that could reduce the read activity? Or is this expected behavior that we need to accept going forward?
I have already tried: Setting File_Cache_Advise to Off — with no effect Upgrading to version 4.0.x — but the high read metrics persist there as well
Is it possible that this is caused by a change in core logic or an upgraded library dependency?
DEV env after upgrade
Here is the config file (we have several of them), but we did not change anything during the upgrade. We will definitely be adjusting some parameters for performance tuning, but for now, I would prefer not to focus on the configuration, and instead understand what exactly changed between the versions that caused such a significant impact on this behavior.
[INPUT]
Name tail
Tag kube.*
Path /var/log/containers/*.log
Path_Key path_filename
Parser docker
DB /var/log/flb_kube.db
Mem_Buf_Limit 3MB
Buffer_Max_Size 1MB
Refresh_Interval 30
[FILTER]
Name kubernetes
Match kube.var.*
Kube_Tag_Prefix kube.var.log.containers.
Merge_Log On
K8S-Logging.Parser On
K8S-Logging.Exclude On
Annotations Off
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_under kubernetes
[FILTER]
Name nest
Match kube.*
Operation lift
Nested_under labels
[FILTER]
Name grep
Match kube.*
Regex app .*
[FILTER]
Name record_modifier
Match kube.*
Allowlist_key app
Allowlist_key namespace_name
Allowlist_key cluster
Allowlist_key log
Allowlist_key message
Allowlist_key pod_name
Allowlist_key host
[FILTER]
name modify
match kube.*
rename namespace_name namespace
rename host hostname
rename pod_name pod
remove_wildcard kubernetes
[OUTPUT]
Name loki
Match *
Host url
Port port
#line_format key_value
line_format json
labels $app, $pod, $cluster, $hostname, $namespace
remove_keys stream, time, logging, keep, docker_id, concat
drop_single_key on
Retry_Limit 1
Thank you very much in advance for your help.