large memory usage increment from about 300M to 10G since v2.1.9 (more than 10 times of v2.1.8)

[hsingli20]

Bug Report

Describe the bug Large memory usage increment from about 300M to 10G since v2.1.9 (more than 10 times of v2.1.8) The memory of container will keep increasing until close to 10G. Memory increases in a sawtooth pattern and then releases. Under the same traffic test and the same configurations for fluentbit, the version v2.1.9 seems introducing such new memory usage pattern, this will cause more memory usage reserved for fluentbit to avoid OOM kill issue.

To Reproduce

• Rubular link if applicable: 100%

• Steps to reproduce the problem:

1. stress the logging traffic
2. monitor the memory usage Expected behavior

Screenshots

Your Environment

• Version used: v2.1.9
• Configuration:

[INPUT]
    name                tail
    tag                 event.kafka.ingress
    alias               kafka.ingress
    buffer_chunk_size   1m
    buffer_max_size     1m
    read_from_head      true
    refresh_interval    5
    rotate_wait         10
    skip_empty_lines    off
    skip_long_lines     true
    key                 message
    db                  /var/log/kafka.ingress.db
    db.sync             normal
    db.locking          true
    db.journal_mode     off
    db.compare_filename true
    path                /var/log/app/*
    exclude_path        /var/log/app/xxx.log,/var/log/app/ingress/*.gz,/var/log/app/ingress/*.tgz
    mem_buf_limit       20MB
    parser              json
    ignore_older        11m

• Environment name and version (e.g. Kubernetes? What version?): kubernetes

[edsiper]

please provide the full configuration including filters and outputs.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.