Getting Error in Fluent bit logs

[sagaarpanjwani]

Running fluent Bit as a sidecar container , with output as kafka

[2022/09/17 18:32:55] [ warn] [engine] chunk '1-1663439562.735639511.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 35 [2022/09/17 18:32:57] [ warn] [engine] failed to flush chunk '1-1663439572.718221096.flb', retry in 8 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 34 [2022/09/17 18:33:01] [ warn] [engine] chunk '1-1663439567.773317947.flb' cannot be retried: task_id=45, input=tail.0 > output=kafka.0 33 [2022/09/17 18:33:02] [ warn] [engine] failed to flush chunk '1-1663439577.704613574.flb', retry in 6 seconds: task_id=40, input=tail.0 > output=kafka.0 (out_id=0) 32 [2022/09/17 18:33:05] [ warn] [engine] chunk '1-1663439572.718221096.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 31 [2022/09/17 18:33:07] [ warn] [engine] failed to flush chunk '1-1663439582.791221059.flb', retry in 9 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 30 [2022/09/17 18:33:08] [ warn] [engine] chunk '1-1663439577.704613574.flb' cannot be retried: task_id=40, input=tail.0 > output=kafka.0 29 [2022/09/17 18:33:12] [ warn] [engine] failed to flush chunk '1-1663439587.768019613.flb', retry in 8 seconds: task_id=40, input=tail.0 > output=kafka.0 (out_id=0) 28 [2022/09/17 18:33:16] [ warn] [engine] chunk '1-1663439582.791221059.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 27 [2022/09/17 18:33:17] [ warn] [engine] failed to flush chunk '1-1663439592.832200141.flb', retry in 9 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 26 [2022/09/17 18:33:20] [ warn] [engine] chunk '1-1663439587.768019613.flb' cannot be retried: task_id=40, input=tail.0 > output=kafka.0 25 [2022/09/17 18:33:22] [ warn] [engine] failed to flush chunk '1-1663439597.708566375.flb', retry in 7 seconds: task_id=40, input=tail.0 > output=kafka.0 (out_id=0) 24 [2022/09/17 18:33:26] [ warn] [engine] chunk '1-1663439592.832200141.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 23 [2022/09/17 18:33:27] [ warn] [engine] failed to flush chunk '1-1663439602.719132176.flb', retry in 7 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 22 [2022/09/17 18:33:29] [ warn] [engine] chunk '1-1663439597.708566375.flb' cannot be retried: task_id=40, input=tail.0 > output=kafka.0 21 [2022/09/17 18:33:32] [ warn] [engine] failed to flush chunk '1-1663439607.700554656.flb', retry in 7 seconds: task_id=40, input=tail.0 > output=kafka.0 (out_id=0) 20 [2022/09/17 18:33:34] [ warn] [engine] chunk '1-1663439602.719132176.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 19 [2022/09/17 18:33:37] [ warn] [engine] failed to flush chunk '1-1663439612.701990545.flb', retry in 8 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 18 [2022/09/17 18:33:39] [ warn] [engine] chunk '1-1663439607.700554656.flb' cannot be retried: task_id=40, input=tail.0 > output=kafka.0 17 [2022/09/17 18:33:42] [ warn] [engine] failed to flush chunk '1-1663439617.704845293.flb', retry in 11 seconds: task_id=40, input=tail.0 > output=kafka.0 (out_id=0) 16 [2022/09/17 18:33:45] [ warn] [engine] chunk '1-1663439612.701990545.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 15 [2022/09/17 18:33:47] [ warn] [engine] failed to flush chunk '1-1663439622.871726418.flb', retry in 9 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 14 [2022/09/17 18:33:52] [ warn] [engine] failed to flush chunk '1-1663439627.712376936.flb', retry in 10 seconds: task_id=45, input=tail.0 > output=kafka.0 (out_id=0) 13 [2022/09/17 18:33:53] [ warn] [engine] chunk '1-1663439617.704845293.flb' cannot be retried: task_id=40, input=tail.0 > output=kafka.0 12 [2022/09/17 18:33:56] [ warn] [engine] chunk '1-1663439622.871726418.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 11 [2022/09/17 18:33:57] [ warn] [engine] failed to flush chunk '1-1663439632.713377006.flb', retry in 7 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 10 [2022/09/17 18:34:02] [ warn] [engine] failed to flush chunk '1-1663439637.784183516.flb', retry in 6 seconds: task_id=40, input=tail.0 > output=kafka.0 (out_id=0) 9 [2022/09/17 18:34:02] [ warn] [engine] chunk '1-1663439627.712376936.flb' cannot be retried: task_id=45, input=tail.0 > output=kafka.0 8 [2022/09/17 18:34:04] [ warn] [engine] chunk '1-1663439632.713377006.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 7 [2022/09/17 18:34:07] [ warn] [engine] failed to flush chunk '1-1663439642.699809701.flb', retry in 8 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 6 [2022/09/17 18:34:08] [ warn] [engine] chunk '1-1663439637.784183516.flb' cannot be retried: task_id=40, input=tail.0 > output=kafka.0 5 [2022/09/17 18:34:12] [ warn] [engine] failed to flush chunk '1-1663439647.723694064.flb', retry in 9 seconds: task_id=40, input=tail.0 > output=kafka.0 (out_id=0) 4 [2022/09/17 18:34:15] [ warn] [engine] chunk '1-1663439642.699809701.flb' cannot be retried: task_id=35, input=tail.0 > output=kafka.0 3 [2022/09/17 18:34:17] [ warn] [engine] failed to flush chunk '1-1663439652.787426671.flb', retry in 11 seconds: task_id=35, input=tail.0 > output=kafka.0 (out_id=0) 2 [2022/09/17 18:34:21] [ warn] [engine] chunk '1-1663439647.723694064.flb' cannot be retried: task_id=40, input=tail.0 > output=kafka.0 1 [2022/09/17 18:34:22] [ warn] [engine] failed to flush chunk '1-1663439657.708165353.flb', retry in 8 seconds: task_id=40, input=tail.0 > output=kafka.0 (out_id=0)

[patrick-stephens]

This looks like it is failing to write to Kafka so please double-check the set up and any credentials are correct. There is not much information here to help so we will need a lot more details on the specific set up you have. Please provide the full Fluent Bit configuration and other information as per the issue template.

Is any output being pushed, i.e. is this just a few failures or is it failing on every write?

What version of Fluent Bit are you using and how are you deploying the side car?

How and what are you deploying? What version of Kafka, etc.? What Kubernetes flavour are you running on, etc.? Can other things push to your Kafka instance?

Run with a debug log_level and please provide the full logs here.

Can you use the debug image to verify the sidecar can connect to your Kafka instance and/or debug any further? There is a Kafka example here as well which may be useful to compare with your set up to see what the deltas are: https://github.com/fluent/fluent-bit/tree/master/examples/kafka_filter

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.