fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard
verified publisher icon fluent

Inconsistent Behavior on DB Management

Open Saya47 opened this issue 1 year ago • 2 comments

Bug Report

Describe the bug If the Tail input plugin is declared before the Systemd input plugin, the DB for Systemd doesn't get created and the same DB used for Tail will be used for Systemd. You can see the DB file opened on my SQLite explorer on the picture below: image

But if you declare the Systemd input plugin first then declare the Tail plugin, then you will have a DB for Systemd and another DB for Tail. You can see the DB files created on the picture below: image

To Reproduce I don't know if this applies to classic config too, but this happened to me on YAML configuration.

fluent-bit.yaml:

includes:
    - env.yaml
    - files.yaml
    - metrics.yaml
    - output.yaml
    - services.yaml
...

files.yaml:

pipeline:
  inputs:
    - Name: tail
      Path: "/var/log/*.log"
      Key: log_raw
      Tag: tail
      # Read_from_Head: True
      DB: /opt/fluent-bit/db/tail.db
...

services.yaml

pipeline:
  inputs:
    - Name: systemd
      Tag: systemd
      DB: /opt/fluent-bit/db/systemd.db
...

Expected behavior If user specifies different DB paths for Tail and Systemd, the user expects the path specified to be used for cursor management, regardless of the order of input plugins declarations.

Your Environment Linux logserver 6.1.0-10-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.37-1 (2023-07-03) x86_64 GNU/Linux

Fluent Bit v2.2.2

Additional context I was running into issues with Systemd with incorrect log timestamps, something similar to what happened here: https://github.com/systemd/systemd/issues/2738

I had some logs on Systemd which were dated for 2025! So When I ran Fluent-bit, it sent all logs from beginning till 2025 and then it stopped sending new logs that were created, due to the issue described on the Systemd issue on link above. This issue with Systemd along the Fluent-bit bug explained on this post caused a lot of confusion.

Thanks very much for reading.

Saya47 avatar Feb 19 '24 06:02 Saya47

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

github-actions[bot] avatar May 21 '24 01:05 github-actions[bot]

any news?

Saya47 avatar May 21 '24 06:05 Saya47

I'm seeing the same behaviour with multiple tail inputs. Again, I'm using the YAML configuration, but my issue seems related to the ordering of the inputs in the config file. For example where my configuration specifies lots of tail inputs with dbs in the following order:

sudo cat /etc/fluent-bit/fluent-bit.yaml | grep db:
    db: "/srv/fluent-bit/rsyslog_tail.db"
    db: "/srv/fluent-bit/icl_rsyslog_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"

Then the databases prefixed with icl_dbcore are not created:

$ sudo ls /srv/fluent-bit/*.db
/srv/fluent-bit/dbcore_report_tail.db
/srv/fluent-bit/dbcore_tail.db
/srv/fluent-bit/icl_rsyslog_tail.db
/srv/fluent-bit/rsyslog_tail.db

However, reordering my configuration such that the usage of the databases is spread out results in them all being created:

sudo cat /etc/fluent-bit/fluent-bit.yaml | grep db:
    db: "/srv/fluent-bit/rsyslog_tail.db"
    db: "/srv/fluent-bit/icl_rsyslog_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"
    db: "/srv/fluent-bit/dbcore_tail.db"
    db: "/srv/fluent-bit/dbcore_report_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_tail.db"
    db: "/srv/fluent-bit/icl_dbcore_report_tail.db"

$ sudo ls -l /srv/fluent-bit/*.db
/srv/fluent-bit/dbcore_report_tail.db
/srv/fluent-bit/dbcore_tail.db
/srv/fluent-bit/icl_dbcore_report_tail.db
/srv/fluent-bit/icl_dbcore_tail.db
/srv/fluent-bit/icl_rsyslog_tail.db
/srv/fluent-bit/rsyslog_tail.db

So it feels like a race condition somewhere...

willholley avatar Jun 13 '24 19:06 willholley