Fluent-bit logs skip issue : scan_blog add(): dismissed

[Ragavendra-Vigneshwaran-R]

I would request some help for following fluent-bit logs which is causing logs skip for our many apps. I have used fluent-bit as a side-car for our app to send the logs to loki. But it is dismiising the logs without sending logs to loki. Would anyone from the team can help here please.. fluent-bit.conf:

` [SERVICE] flush 1 daemon Off log_level debug http_server Off http_listen 0.0.0.0 http_port 2020 storage.metrics on

[INPUT]
    Name        tail
    Path        /home/rundeck/var/logs/rundeck/*/job/*/logs/*.rdlog
    Tag         rundeck_logs
    Refresh_Interval 5

[OUTPUT]
    name  stdout
    match *

[OUTPUT]
    Name        loki
    Match       *
    Host        loki.loki
    port        3100
    labels job=rundeck-execution-logs`

Error logs:

`Fluent Bit v2.2.0

• Copyright (C) 2015-2023 The Fluent Bit Authors
• Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
• https://fluentbit.io

[2023/11/23 09:45:12] [ info] Configuration: [2023/11/23 09:45:12] [ info] flush time | 1.000000 seconds [2023/11/23 09:45:12] [ info] grace | 5 seconds [2023/11/23 09:45:12] [ info] daemon | 0 [2023/11/23 09:45:12] [ info] ___________ [2023/11/23 09:45:12] [ info] inputs: [2023/11/23 09:45:12] [ info] tail [2023/11/23 09:45:12] [ info] ___________ [2023/11/23 09:45:12] [ info] filters: [2023/11/23 09:45:12] [ info] ___________ [2023/11/23 09:45:12] [ info] outputs: [2023/11/23 09:45:12] [ info] stdout.0 [2023/11/23 09:45:12] [ info] loki.1 [2023/11/23 09:45:12] [ info] ___________ [2023/11/23 09:45:12] [ info] collectors: [2023/11/23 09:45:12] [ info] [fluent bit] version=2.2.0, commit=db8487d123, pid=1 [2023/11/23 09:45:12] [debug] [engine] coroutine stack size: 24576 bytes (24.0K) [2023/11/23 09:45:12] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128 [2023/11/23 09:45:12] [ info] [cmetrics] version=0.6.4 [2023/11/23 09:45:12] [ info] [ctraces ] version=0.3.1 [2023/11/23 09:45:12] [ info] [input:tail:tail.0] initializing [2023/11/23 09:45:12] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only) [2023/11/23 09:45:12] [debug] [tail:tail.0] created event channels: read=21 write=22 [2023/11/23 09:45:12] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input [2023/11/23 09:45:12] [debug] [input:tail:tail.0] inotify watch fd=27 [2023/11/23 09:45:12] [debug] [input:tail:tail.0] scanning path /home/rundeck/var/logs/rundeck//job//logs/.rdlog [2023/11/23 09:45:12] [debug] [input:tail:tail.0] inode=20038348 with offset=0 appended as /home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog [2023/11/23 09:45:12] [debug] [input:tail:tail.0] scan_glob add(): /home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog, inode 20038348 [2023/11/23 09:45:12] [debug] [input:tail:tail.0] 1 new files found on path '/home/rundeck/var/logs/rundeck//job//logs/.rdlog' [2023/11/23 09:45:12] [debug] [stdout:stdout.0] created event channels: read=29 write=30 [2023/11/23 09:45:12] [debug] [loki:loki.1] created event channels: read=36 write=37 [2023/11/23 09:45:12] [ info] [output:loki:loki.1] configured, hostname=loki.loki:3100 [2023/11/23 09:45:12] [debug] [router] match rule tail.0:stdout.0 [2023/11/23 09:45:12] [debug] [router] match rule tail.0:loki.1 [2023/11/23 09:45:12] [ info] [output:stdout:stdout.0] worker #0 started [2023/11/23 09:45:12] [ info] [sp] stream processor started [2023/11/23 09:45:12] [debug] [input chunk] update output instances with new chunk size diff=42, records=1, input=tail.0 [2023/11/23 09:45:12] [debug] [input:tail:tail.0] [static files] processed 8b [2023/11/23 09:45:12] [debug] [input:tail:tail.0] inode=20038348 file=/home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog promote to TAIL_EVENT [2023/11/23 09:45:12] [ info] [input:tail:tail.0] inotify_fs_add(): inode=20038348 watch_fd=1 name=/home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog [2023/11/23 09:45:12] [debug] [input:tail:tail.0] [static files] processed 0b, done [2023/11/23 09:45:13] [debug] [task] created task=0x7fe4ea436aa0 id=0 OK [2023/11/23 09:45:13] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0 [0] rundeck_logs: [[1700732712.193364756, {}], {"log"=>"testing"}] [2023/11/23 09:45:13] [debug] [out flush] cb_destroy coro_id=0 [2023/11/23 09:45:13] [debug] [upstream] KA connection #51 to loki.loki:3100 is connected [2023/11/23 09:45:13] [debug] [http_client] not using http_proxy for header [2023/11/23 09:45:13] [debug] [output:loki:loki.1] loki.loki:3100, HTTP status=204 [2023/11/23 09:45:13] [debug] [upstream] KA connection #51 to loki.loki:3100 is now available [2023/11/23 09:45:13] [debug] [out flush] cb_destroy coro_id=0 [2023/11/23 09:45:13] [debug] [task] destroy task=0x7fe4ea436aa0 (task_id=0) [2023/11/23 09:45:17] [debug] [input:tail:tail.0] scanning path /home/rundeck/var/logs/rundeck//job//logs/.rdlog [2023/11/23 09:45:17] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog, inode 20038348 [2023/11/23 09:45:17] [debug] [input:tail:tail.0] 0 new files found on path '/home/rundeck/var/logs/rundeck//job//logs/.rdlog' [2023/11/23 09:45:22] [debug] [input:tail:tail.0] scanning path /home/rundeck/var/logs/rundeck//job//logs/.rdlog [2023/11/23 09:45:22] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog, inode 20038348 [2023/11/23 09:45:22] [debug] [input:tail:tail.0] 0 new files found on path '/home/rundeck/var/logs/rundeck//job//logs/.rdlog' [2023/11/23 09:45:27] [debug] [input:tail:tail.0] scanning path /home/rundeck/var/logs/rundeck//job//logs/*.rdlog [2023/11/23 09:45:27] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog, inode 20038348`

[patrick-stephens]

It looks like it has got data from the file, prior to Loki connection being available but it should retry later.

file=/home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog promote to TAIL_EVENT
[2023/11/23 09:45:12] [ info] [input:tail:tail.0] inotify_fs_add(): inode=20038348 watch_fd=1 name=/home/rundeck/var/logs/rundeck/sample/job/sample/logs/s1.rdlog
[2023/11/23 09:45:12] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/11/23 09:45:13] [debug] [task] created task=0x7fe4ea436aa0 id=0 OK
[2023/11/23 09:45:13] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[0] rundeck_logs: [[1700732712.193364756, {}], {"log"=>"testing"}]
[2023/11/23 09:45:13] [debug] [out flush] cb_destroy coro_id=0
[2023/11/23 09:45:13] [debug] [upstream] KA connection https://github.com/fluent/fluent-bit/issues/51 to loki.loki:3100 is connected
[2023/11/23 09:45:13] [debug] [http_client] not using http_proxy for header
[2023/11/23 09:45:13] [debug] [output:loki:loki.1] loki.loki:3100, HTTP status=204
[2023/11/23 09:45:13] [debug] [upstream] KA connection https://github.com/fluent/fluent-bit/issues/51 to loki.loki:3100 is now available

After that, are you updating the file otherwise there is no new data to read so it should be dismissed?

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.