fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

High filesystem read activity after upgrade without corresponding throughput increase

Open hoooli opened this issue 4 months ago • 0 comments

After upgrading from Fluent Bit version 3.0.7 to 3.1.2, I noticed a significant increase in the container_fs_reads_bytes_total metric. With version 3.0.7, values typically ranged from tens to hundreds of kilobytes. After upgrading — with no changes in configuration — the minimum values are now in megabytes, and the average is in the tens of MB.

In our production environment, this triggers alerts under higher load, where many more containers are running. The difference in filesystem read activity between the versions is therefore very large and concerning.

I would like to ask: Which specific change or commit could have introduced this behavior? Is there a parameter or setting that could reduce the read activity? Or is this expected behavior that we need to accept going forward?

I have already tried: Setting File_Cache_Advise to Off — with no effect Upgrading to version 4.0.x — but the high read metrics persist there as well

Is it possible that this is caused by a change in core logic or an upgraded library dependency?

DEV env after upgrade Image

Here is the config file (we have several of them), but we did not change anything during the upgrade. We will definitely be adjusting some parameters for performance tuning, but for now, I would prefer not to focus on the configuration, and instead understand what exactly changed between the versions that caused such a significant impact on this behavior.

    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        Path_Key          path_filename
        Parser            docker
        DB                /var/log/flb_kube.db
        Mem_Buf_Limit     3MB
        Buffer_Max_Size   1MB
        Refresh_Interval  30
    [FILTER]
        Name                kubernetes
        Match               kube.var.*
        Kube_Tag_Prefix     kube.var.log.containers.
        Merge_Log           On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Annotations         Off
    [FILTER]
        Name         nest
        Match        kube.*
        Operation    lift
        Nested_under kubernetes
    [FILTER]
        Name         nest
        Match        kube.*
        Operation    lift
        Nested_under labels
    [FILTER]
        Name         grep
        Match        kube.*
        Regex        app .*
    [FILTER]
        Name         record_modifier
        Match        kube.*
        Allowlist_key app
        Allowlist_key namespace_name
        Allowlist_key cluster
        Allowlist_key log
        Allowlist_key message
        Allowlist_key pod_name
        Allowlist_key host
    [FILTER]
        name                  modify
        match                 kube.*
        rename                namespace_name namespace
        rename                host hostname
        rename                pod_name pod
        remove_wildcard       kubernetes
    [OUTPUT]
        Name        loki
        Match       *
        Host        url
        Port        port
        #line_format key_value
        line_format json
        labels      $app, $pod, $cluster, $hostname, $namespace
        remove_keys stream, time, logging, keep, docker_id, concat
        drop_single_key on
        Retry_Limit  1

Thank you very much in advance for your help.

hoooli avatar Jun 18 '25 13:06 hoooli