Tail plugin reading incomplete log lines upon rotation

[rashmichandrashekar]

Bug Report

Describe the bug After the log file is rotated, the tail plugin seems to read the log line from the middle of the line, which leads to parsing errors.

To Reproduce Have the tail plugin tail files that will be rotated and you will notice missing log lines from the beginning of the new file

• Example log message if applicable: In our case, since we try to parse the timestamp and the log line that is read is somewhere in the middle, the time stamp parsing fails - Error while converting logEntryTimeStamp for telemetry purposes: parsing time "22-04-22T08:03:24.345601269Z" as "2006-01-02T15:04:05Z07:00": cannot parse "4-22T08:03:24.345601269Z" as "2006"

**Expected behavior**
Fluent bit tail plugin should tail the log lines from the beginning of each log entry.

**Your Environment**
The environment is a linux container in an AKS pod. We run fluentbit as a process and tail log files to send them to Azure Log Analytics workspace

* Version used: v1.7.8
* Configuration:
[SERVICE]
    #Default service flush interval is 15 seconds
    Flush         15
    HTTP_Server   Off
    Daemon        Off
    storage.path  /var/opt/microsoft/docker-cimprov/state/flbstore/
    storage.sync  normal
    storage.checksum off
    storage.backlog.mem_limit 10M
    Log_Level     info
    Parsers_File  /etc/opt/microsoft/docker-cimprov/azm-containers-parser.conf
    Log_File      /var/opt/microsoft/docker-cimprov/log/fluent-bit.log

[INPUT]
    Name tail
    Tag oms.container.log.la.*
    Path /var/log/containers/*.log
    Read_from_Head true
    DB /var/log/omsagent-fblogs.db
    DB.Sync Off
    Parser cri
    Mem_Buf_Limit 10m
    Buffer_Chunk_Size 1m
    Buffer_Max_Size 1m
    Rotate_Wait 20
    Refresh_Interval 30
    Path_Key filepath
    Skip_Long_Lines On
    Ignore_Older 5m

* Environment name and version (e.g. Kubernetes? What version?): kubernetes(AKS), 1.22.6

* Operating System and version: Ubuntu 18.04
* Filters and plugins: Tail plugin

**Additional context**
<!--- How has this issue affected you? What are you trying to accomplish? -->
Leads to missing log lines in the log collection solution we offer customers

[ganga1980]

/track @ganga1980

[patrick-stephens]

Does it happen with more recent versions, e.g. 1.8.15 and 1.9.3?

[rashmichandrashekar]

@patrick-stephens - Yes, this happens with 1.9.3 as well.

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.

[liguangcheng]

@rashmichandrashekar I also faced this issue, the root cause is fluent bit use the inode to distinguish new and old file, when a file use one inode to record postition in sqlite, once the inode allocate for another new file, the new file will be read from the position with the record in sqlit that belong the a old file, so the new file content could not be complete