FluentBit writes all records to ouput everytime source file is modified

[olespagnon-vdm]

Bug Report

Describe the bug Using Tail input result in duplicated records in output.

To Reproduce

• Rubular link : https://rubular.com/r/EErpPTcOeW4j0p
• Example log message if applicable:

2024-10-15 05:30:59.436 []  INF XXX.YYYYYY.BasketService.Api.Program->Main():47     Starting host

• Steps to reproduce the problem:

1. Start fluent-bit
2. Edit the log file once
3. Check the output from fluent bit
4. Edit the log file a second time
5. See the fluent bit append the whole log file again in the output

Expected behavior Fluent-bit should only add new lines to the output when the source is edited

Your Environment

• Version used: v3.1.9
• Configuration:

[SERVICE]
    Daemon Off
    Flush 1
    Log_Level debug
    Parsers_File ./parsers.conf
    db test.db

[INPUT]
    Name tail
    Path ./*.log
    Path_Key filename
    Parser msp-logs
    Tag app_log
    
[FILTER]
    Name parser
    Key_Name filename
    Parser log_name_parser
    Reserve_Data True
    Match app_log

[OUTPUT]
    Name file
    Path ./test
    Mkdir True

Parsers :

[PARSER]
    Name log_name_parser
    Format regex
    Key_Name filename
    Regex (?<logger>[a-zA-Z]+\.[a-zA-Z]+)\d+\.log
[PARSER]
    Name msp-logs
    Format regex
    Regex (?<date>\d{4}-\d{2}-\d{2}\s+\d{2}:\d{2}:\d{2}).\d{3}\s+\[\]\s+(?<log_lvl>INF|DBG|ERR|WAR)\s+(?<event_source>\S+)\s+(?<log_msg>.+)

• Operating System and version: Ubuntu

Additional context Even when using the DB option, no DB file is created, but even without it the Tail output every record twice

[patrick-stephens]

My guess is something is not right somewhere in your system - are you sure the config is being applied correctly and are you restarting Fluent Bit? If there is no DB file then that's why I suspect you've got a config problem, you should have.

[olespagnon-vdm]

Hello, thanks for the answer. I restart Fluent for each test. Each time I delete the output file from Fluent. The config seems to be applied as it writes correctly to the ouput file following the changes in formatting I make. I've also reinstalled Fluent to be sure. Maybe there is something i'm missing. What should I check ?

[olespagnon-vdm]

Ok, to give a little more context. I was adding lines manually to my log file. I've created a bash script to automatically log to a file. This process doesn't duplicate line. However, I still don't see a DB file. The mistery thickens...

[olespagnon-vdm]

I'm closing this issue. Deploying it to an EKS cluster solved it somehow. Still now idea why it did not work locally