tail input plugin /var/log/containers/*.log doesnt push logs to destination

[tanmaya-panda1]

Bug Report

Describe the bug I am trying to push kubernetes logs to kusto as destination. tail input plugin /var/log/containers/*.log doesn't seem to work but it works when i use /var/log/containers/fluent*.log. I have added the volume mounts as well to avoid the symlinks issue.

I am using helm chart with the following config

helm repo add fluent https://fluent.github.io/helmcharts

helm repo update helm install fluent-bit-demo fluent/fluent-bit --values kusto-values-template.yaml

The contents of kusto-values-template.yaml is as follows

logLevel: trace

kind: DaemonSet

image: repository: fluentbitcontainereast.azurecr.io/fluent-bit-kusto tag: 2.1.9 pullPolicy: Always

config: service: | [SERVICE] Daemon Off Flush 1 Log_Level trace HTTP_Server On HTTP_Listen 0.0.0.0 Parsers_File parsers.conf Parsers_File custom_parsers.conf HTTP_Port 2020 Health_Check On

inputs: | [INPUT] Name tail Path /var/log/containers/*.log multiline.parser docker, cri Tag kube.* Mem_Buf_Limit 1MB Skip_Long_Lines Off Refresh_Interval 10

filters: | [FILTER] Name kubernetes Match * Merge_Log On Merge_Log_key log_processed K8S-Logging.Parser On K8S-Logging.Exclude Off

outputs: | [OUTPUT] name azure_kusto match * tenant_id xxxxx client_id xxxxx client_secret xxxxx ingestion_endpoint xxxxx database_name xxxx table_name xxxx ingestion_mapping_reference xxxx

customParsers: | [PARSER] Name docker_no_time Format json Time_Keep On Time_Key time Time_Format %Y-%m-%dT%H:%M:%S.%L

volumeMounts:

• name: config mountPath: /fluent-bit/etc/conf
• name: var-log-containers mountPath: /var/log/containers
• name: var-log-pods mountPath: /var/log/pods
• name: var-lib-docker-containers mountPath: /var/lib/docker/containers

daemonSetVolumes:

• name: varlog hostPath: path: /var/log/containers
• name: varpods hostPath: path: /var/log/pods
• name: varlibdockercontainers hostPath: path: /var/lib/docker/containers
• name: etcmachineid hostPath: path: /etc/machine-id type: File

daemonSetVolumeMounts:

• name: varlog mountPath: /var/log/containers
• name: varpods mountPath: /var/log/pods
• name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true
• name: etcmachineid mountPath: /etc/machine-id readOnly: true

To Reproduce

• Rubular link if applicable:
• Example log message if applicable:

Fluent Bit v2.2.1
* Copyright (C) 2015-2023 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/11/24 21:21:27] [ info] Configuration:
[2023/11/24 21:21:27] [ info]  flush time     | 1.000000 seconds
[2023/11/24 21:21:27] [ info]  grace          | 5 seconds
[2023/11/24 21:21:27] [ info]  daemon         | 0
[2023/11/24 21:21:27] [ info] ___________
[2023/11/24 21:21:27] [ info]  inputs:
[2023/11/24 21:21:27] [ info]      tail
[2023/11/24 21:21:27] [ info] ___________
[2023/11/24 21:21:27] [ info]  filters:
[2023/11/24 21:21:27] [ info]      kubernetes.0
[2023/11/24 21:21:27] [ info] ___________
[2023/11/24 21:21:27] [ info]  outputs:
[2023/11/24 21:21:27] [ info]      azure_kusto.0
[2023/11/24 21:21:27] [ info] ___________
[2023/11/24 21:21:27] [ info]  collectors:
[2023/11/24 21:21:27] [ info] [fluent bit] version=2.2.1, commit=8733b25df1, pid=1
[2023/11/24 21:21:27] [debug] [engine] coroutine stack size: 24576 bytes (24.0K)
[2023/11/24 21:21:27] [ info] [storage] ver=1.5.1, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/11/24 21:21:27] [ info] [cmetrics] version=0.6.5
[2023/11/24 21:21:27] [ info] [ctraces ] version=0.3.1
[2023/11/24 21:21:27] [ info] [input:tail:tail.0] initializing
[2023/11/24 21:21:27] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2023/11/24 21:21:27] [debug] [tail:tail.0] created event channels: read=21 write=22
[2023/11/24 21:21:27] [ info] [input:tail:tail.0] multiline core started
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] flb_tail_fs_inotify_init() initializing inotify tail input
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inotify watch fd=28
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1813411 with offset=386852 appended as /var/log/containers/ama-logs-d668n_kube-system_addon-token-adapter-322f5b5f1cbb9a10f662da26ca34c3a29047b0947b5bda87ef01a7990eb43eb4.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/ama-logs-d668n_kube-system_addon-token-adapter-322f5b5f1cbb9a10f662da26ca34c3a29047b0947b5bda87ef01a7990eb43eb4.log, inode 1813411
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1813459 with offset=8861 appended as /var/log/containers/ama-logs-d668n_kube-system_ama-logs-bf6949ab252b9ceda2d9023aa1c293535c87e592f5022a886d5fd52f0af72eb5.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/ama-logs-d668n_kube-system_ama-logs-bf6949ab252b9ceda2d9023aa1c293535c87e592f5022a886d5fd52f0af72eb5.log, inode 1813459
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1813497 with offset=4591 appended as /var/log/containers/ama-logs-d668n_kube-system_ama-logs-prometheus-75d72411158111544c959d11394a9c804495b7c3617a4eb0db39e1bdba9b8b44.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/ama-logs-d668n_kube-system_ama-logs-prometheus-75d72411158111544c959d11394a9c804495b7c3617a4eb0db39e1bdba9b8b44.log, inode 1813497
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1813431 with offset=337024 appended as /var/log/containers/ama-metrics-node-tvs5r_kube-system_addon-token-adapter-83fab8b75208fab94ca166bbb4a154b83734be35521c609334722d60d72c2fdd.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/ama-metrics-node-tvs5r_kube-system_addon-token-adapter-83fab8b75208fab94ca166bbb4a154b83734be35521c609334722d60d72c2fdd.log, inode 1813431
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1810243 with offset=9808 appended as /var/log/containers/ama-metrics-node-tvs5r_kube-system_prometheus-collector-e3de35fc24f6e5e7b0f50a87468521f98a5e327caf5ed91f71a8f452884c1b5e.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/ama-metrics-node-tvs5r_kube-system_prometheus-collector-e3de35fc24f6e5e7b0f50a87468521f98a5e327caf5ed91f71a8f452884c1b5e.log, inode 1810243
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1806932 with offset=4268058 appended as /var/log/containers/azure-ip-masq-agent-89vgs_kube-system_azure-ip-masq-agent-81675abb5b05bf9588ba028c9641c051e2f61c1342317358ac8ab46988e2df0e.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/azure-ip-masq-agent-89vgs_kube-system_azure-ip-masq-agent-81675abb5b05bf9588ba028c9641c051e2f61c1342317358ac8ab46988e2df0e.log, inode 1806932
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1814633 with offset=74955 appended as /var/log/containers/calico-kube-controllers-7957ff69b4-j4r28_calico-system_calico-kube-controllers-100385987bdaf0dad011964030ec8018f65f625b08af1e788a35500b75144125.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/calico-kube-controllers-7957ff69b4-j4r28_calico-system_calico-kube-controllers-100385987bdaf0dad011964030ec8018f65f625b08af1e788a35500b75144125.log, inode 1814633
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807538 with offset=3164510 appended as /var/log/containers/calico-node-56c7v_calico-system_calico-node-557fd3750d680c5b05b44cdc5d3ebf4972aeeeddd18c79744d805e13eaa3ec45.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/calico-node-56c7v_calico-system_calico-node-557fd3750d680c5b05b44cdc5d3ebf4972aeeeddd18c79744d805e13eaa3ec45.log, inode 1807538
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807037 with offset=0 appended as /var/log/containers/calico-node-56c7v_calico-system_flexvol-driver-54be58e8799a9af362ce04017e36a19b3e31f70e98a04a4ca25ca691c9d3b775.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/calico-node-56c7v_calico-system_flexvol-driver-54be58e8799a9af362ce04017e36a19b3e31f70e98a04a4ca25ca691c9d3b775.log, inode 1807037
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807087 with offset=1003 appended as /var/log/containers/cloud-node-manager-6p8nd_kube-system_cloud-node-manager-90eec4734379b7f22a84964060b74991a16abd8e88ba62319d19aa7ffc03be25.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/cloud-node-manager-6p8nd_kube-system_cloud-node-manager-90eec4734379b7f22a84964060b74991a16abd8e88ba62319d19aa7ffc03be25.log, inode 1807087
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807261 with offset=12043 appended as /var/log/containers/csi-azuredisk-node-xhtjv_kube-system_azuredisk-ac97724d4596642ef35c1d2d0c576568312baf10786a6b9fda40e7eed9edcbb7.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/csi-azuredisk-node-xhtjv_kube-system_azuredisk-ac97724d4596642ef35c1d2d0c576568312baf10786a6b9fda40e7eed9edcbb7.log, inode 1807261
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807180 with offset=368 appended as /var/log/containers/csi-azuredisk-node-xhtjv_kube-system_liveness-probe-0053c18fa4754385a806000f15ceab7f2ee64c167b06d2a182161cfb5ae7deb5.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/csi-azuredisk-node-xhtjv_kube-system_liveness-probe-0053c18fa4754385a806000f15ceab7f2ee64c167b06d2a182161cfb5ae7deb5.log, inode 1807180
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807212 with offset=1586 appended as /var/log/containers/csi-azuredisk-node-xhtjv_kube-system_node-driver-registrar-d704be2678e43eb22ca8b4cca4dfe65e010b5ef73055cd6a1d83a63d0686f8d6.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/csi-azuredisk-node-xhtjv_kube-system_node-driver-registrar-d704be2678e43eb22ca8b4cca4dfe65e010b5ef73055cd6a1d83a63d0686f8d6.log, inode 1807212
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807264 with offset=12609 appended as /var/log/containers/csi-azurefile-node-kfxkd_kube-system_azurefile-da1ed79280215f04afef0531afcb04a61ab3c466fe2dda6725cef7b7b8844293.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/csi-azurefile-node-kfxkd_kube-system_azurefile-da1ed79280215f04afef0531afcb04a61ab3c466fe2dda6725cef7b7b8844293.log, inode 1807264
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807138 with offset=368 appended as /var/log/containers/csi-azurefile-node-kfxkd_kube-system_liveness-probe-e960a07123f3ce8e9b5625832ee21c416bb84b3ba87c588015fce18ba3b04054.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/csi-azurefile-node-kfxkd_kube-system_liveness-probe-e960a07123f3ce8e9b5625832ee21c416bb84b3ba87c588015fce18ba3b04054.log, inode 1807138
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807211 with offset=1584 appended as /var/log/containers/csi-azurefile-node-kfxkd_kube-system_node-driver-registrar-db225223c3baee655dcfb492ea14e85caace41e209fa32c364d2c941023bc964.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/csi-azurefile-node-kfxkd_kube-system_node-driver-registrar-db225223c3baee655dcfb492ea14e85caace41e209fa32c364d2c941023bc964.log, inode 1807211
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=4128853 with offset=11142 appended as /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, inode 4128853
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1807489 with offset=1127548 appended as /var/log/containers/kube-proxy-ssrlq_kube-system_kube-proxy-6d86c3689c16d8c7a9bdff2e42982f8f9bed657303457593fcf5ae58ab581b4e.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/kube-proxy-ssrlq_kube-system_kube-proxy-6d86c3689c16d8c7a9bdff2e42982f8f9bed657303457593fcf5ae58ab581b4e.log, inode 1807489
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] inode=1806977 with offset=299 appended as /var/log/containers/kube-proxy-ssrlq_kube-system_kube-proxy-bootstrap-b6a0314e1611fbe649d960720cd93f2742362eb2241d9241605902dbe0537d0b.log
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] scan_glob add(): /var/log/containers/kube-proxy-ssrlq_kube-system_kube-proxy-bootstrap-b6a0314e1611fbe649d960720cd93f2742362eb2241d9241605902dbe0537d0b.log, inode 1806977
[2023/11/24 21:21:27] [debug] [input:tail:tail.0] 19 new files found on path '/var/log/containers/*.log'
[2023/11/24 21:21:27] [ info] [filter:kubernetes:kubernetes.0] https=1 host=kubernetes.default.svc port=443
[2023/11/24 21:21:27] [ info] [filter:kubernetes:kubernetes.0]  token updated
[2023/11/24 21:21:27] [ info] [filter:kubernetes:kubernetes.0] local POD info OK
[2023/11/24 21:21:32] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with API server...
[2023/11/24 21:21:32] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information
[2023/11/24 21:21:32] [debug] [http_client] not using http_proxy for header
[2023/11/24 21:21:32] [debug] [http_client] server kubernetes.default.svc:443 will close connection #48
[2023/11/24 21:21:32] [debug] [filter:kubernetes:kubernetes.0] Request (ns=default, pod=fluent-bit-demo-r9bj5) http_do=0, HTTP Status: 200
[2023/11/24 21:21:32] [ info] [filter:kubernetes:kubernetes.0] connectivity OK
[2023/11/24 21:21:32] [debug] [azure_kusto:azure_kusto.0] created event channels: read=48 write=49
[2023/11/24 21:21:32] [ info] [output:azure_kusto:azure_kusto.0] endpoint='https://ingest-sdkse2etest.eastus.kusto.windows.net', database='e2e', table='FluentBitTemp'
[2023/11/24 21:21:32] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2023/11/24 21:21:32] [ info] [sp] stream processor started
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1813411 file=/var/log/containers/ama-logs-d668n_kube-system_addon-token-adapter-322f5b5f1cbb9a10f662da26ca34c3a29047b0947b5bda87ef01a7990eb43eb4.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1813411 watch_fd=1 name=/var/log/containers/ama-logs-d668n_kube-system_addon-token-adapter-322f5b5f1cbb9a10f662da26ca34c3a29047b0947b5bda87ef01a7990eb43eb4.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1813459 file=/var/log/containers/ama-logs-d668n_kube-system_ama-logs-bf6949ab252b9ceda2d9023aa1c293535c87e592f5022a886d5fd52f0af72eb5.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1813459 watch_fd=2 name=/var/log/containers/ama-logs-d668n_kube-system_ama-logs-bf6949ab252b9ceda2d9023aa1c293535c87e592f5022a886d5fd52f0af72eb5.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1813497 file=/var/log/containers/ama-logs-d668n_kube-system_ama-logs-prometheus-75d72411158111544c959d11394a9c804495b7c3617a4eb0db39e1bdba9b8b44.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1813497 watch_fd=3 name=/var/log/containers/ama-logs-d668n_kube-system_ama-logs-prometheus-75d72411158111544c959d11394a9c804495b7c3617a4eb0db39e1bdba9b8b44.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1813431 file=/var/log/containers/ama-metrics-node-tvs5r_kube-system_addon-token-adapter-83fab8b75208fab94ca166bbb4a154b83734be35521c609334722d60d72c2fdd.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1813431 watch_fd=4 name=/var/log/containers/ama-metrics-node-tvs5r_kube-system_addon-token-adapter-83fab8b75208fab94ca166bbb4a154b83734be35521c609334722d60d72c2fdd.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1810243 file=/var/log/containers/ama-metrics-node-tvs5r_kube-system_prometheus-collector-e3de35fc24f6e5e7b0f50a87468521f98a5e327caf5ed91f71a8f452884c1b5e.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1810243 watch_fd=5 name=/var/log/containers/ama-metrics-node-tvs5r_kube-system_prometheus-collector-e3de35fc24f6e5e7b0f50a87468521f98a5e327caf5ed91f71a8f452884c1b5e.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1806932 file=/var/log/containers/azure-ip-masq-agent-89vgs_kube-system_azure-ip-masq-agent-81675abb5b05bf9588ba028c9641c051e2f61c1342317358ac8ab46988e2df0e.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806932 watch_fd=6 name=/var/log/containers/azure-ip-masq-agent-89vgs_kube-system_azure-ip-masq-agent-81675abb5b05bf9588ba028c9641c051e2f61c1342317358ac8ab46988e2df0e.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1814633 file=/var/log/containers/calico-kube-controllers-7957ff69b4-j4r28_calico-system_calico-kube-controllers-100385987bdaf0dad011964030ec8018f65f625b08af1e788a35500b75144125.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1814633 watch_fd=7 name=/var/log/containers/calico-kube-controllers-7957ff69b4-j4r28_calico-system_calico-kube-controllers-100385987bdaf0dad011964030ec8018f65f625b08af1e788a35500b75144125.log
[2023/11/24 21:21:32] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information
[2023/11/24 21:21:32] [debug] [http_client] not using http_proxy for header
[2023/11/24 21:21:32] [debug] [http_client] server kubernetes.default.svc:443 will close connection #71
[2023/11/24 21:21:32] [debug] [filter:kubernetes:kubernetes.0] Request (ns=calico-system, pod=calico-node-56c7v) http_do=0, HTTP Status: 200
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=33104, records=34, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807037 file=/var/log/containers/calico-node-56c7v_calico-system_flexvol-driver-54be58e8799a9af362ce04017e36a19b3e31f70e98a04a4ca25ca691c9d3b775.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807037 watch_fd=8 name=/var/log/containers/calico-node-56c7v_calico-system_flexvol-driver-54be58e8799a9af362ce04017e36a19b3e31f70e98a04a4ca25ca691c9d3b775.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807087 file=/var/log/containers/cloud-node-manager-6p8nd_kube-system_cloud-node-manager-90eec4734379b7f22a84964060b74991a16abd8e88ba62319d19aa7ffc03be25.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807087 watch_fd=9 name=/var/log/containers/cloud-node-manager-6p8nd_kube-system_cloud-node-manager-90eec4734379b7f22a84964060b74991a16abd8e88ba62319d19aa7ffc03be25.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807261 file=/var/log/containers/csi-azuredisk-node-xhtjv_kube-system_azuredisk-ac97724d4596642ef35c1d2d0c576568312baf10786a6b9fda40e7eed9edcbb7.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807261 watch_fd=10 name=/var/log/containers/csi-azuredisk-node-xhtjv_kube-system_azuredisk-ac97724d4596642ef35c1d2d0c576568312baf10786a6b9fda40e7eed9edcbb7.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807180 file=/var/log/containers/csi-azuredisk-node-xhtjv_kube-system_liveness-probe-0053c18fa4754385a806000f15ceab7f2ee64c167b06d2a182161cfb5ae7deb5.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807180 watch_fd=11 name=/var/log/containers/csi-azuredisk-node-xhtjv_kube-system_liveness-probe-0053c18fa4754385a806000f15ceab7f2ee64c167b06d2a182161cfb5ae7deb5.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807212 file=/var/log/containers/csi-azuredisk-node-xhtjv_kube-system_node-driver-registrar-d704be2678e43eb22ca8b4cca4dfe65e010b5ef73055cd6a1d83a63d0686f8d6.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807212 watch_fd=12 name=/var/log/containers/csi-azuredisk-node-xhtjv_kube-system_node-driver-registrar-d704be2678e43eb22ca8b4cca4dfe65e010b5ef73055cd6a1d83a63d0686f8d6.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807264 file=/var/log/containers/csi-azurefile-node-kfxkd_kube-system_azurefile-da1ed79280215f04afef0531afcb04a61ab3c466fe2dda6725cef7b7b8844293.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807264 watch_fd=13 name=/var/log/containers/csi-azurefile-node-kfxkd_kube-system_azurefile-da1ed79280215f04afef0531afcb04a61ab3c466fe2dda6725cef7b7b8844293.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807138 file=/var/log/containers/csi-azurefile-node-kfxkd_kube-system_liveness-probe-e960a07123f3ce8e9b5625832ee21c416bb84b3ba87c588015fce18ba3b04054.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807138 watch_fd=14 name=/var/log/containers/csi-azurefile-node-kfxkd_kube-system_liveness-probe-e960a07123f3ce8e9b5625832ee21c416bb84b3ba87c588015fce18ba3b04054.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807211 file=/var/log/containers/csi-azurefile-node-kfxkd_kube-system_node-driver-registrar-db225223c3baee655dcfb492ea14e85caace41e209fa32c364d2c941023bc964.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807211 watch_fd=15 name=/var/log/containers/csi-azurefile-node-kfxkd_kube-system_node-driver-registrar-db225223c3baee655dcfb492ea14e85caace41e209fa32c364d2c941023bc964.log
[2023/11/24 21:21:32] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information
[2023/11/24 21:21:32] [debug] [http_client] not using http_proxy for header
[2023/11/24 21:21:32] [debug] [http_client] server kubernetes.default.svc:443 will close connection #71
[2023/11/24 21:21:32] [debug] [filter:kubernetes:kubernetes.0] Request (ns=default, pod=fluent-bit-demo-r9bj5) http_do=0, HTTP Status: 200
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=53110, records=54, input=tail.0
[2023/11/24 21:21:32] [debug] [filter:kubernetes:kubernetes.0] Send out request to API Server for pods information
[2023/11/24 21:21:32] [debug] [http_client] not using http_proxy for header
[2023/11/24 21:21:32] [debug] [http_client] server kubernetes.default.svc:443 will close connection #71
[2023/11/24 21:21:32] [debug] [filter:kubernetes:kubernetes.0] Request (ns=kube-system, pod=kube-proxy-ssrlq) http_do=0, HTTP Status: 200
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=8943, records=11, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1806977 file=/var/log/containers/kube-proxy-ssrlq_kube-system_kube-proxy-bootstrap-b6a0314e1611fbe649d960720cd93f2742362eb2241d9241605902dbe0537d0b.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1806977 watch_fd=16 name=/var/log/containers/kube-proxy-ssrlq_kube-system_kube-proxy-bootstrap-b6a0314e1611fbe649d960720cd93f2742362eb2241d9241605902dbe0537d0b.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 21.3K
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807538 file=/var/log/containers/calico-node-56c7v_calico-system_calico-node-557fd3750d680c5b05b44cdc5d3ebf4972aeeeddd18c79744d805e13eaa3ec45.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807538 watch_fd=17 name=/var/log/containers/calico-node-56c7v_calico-system_calico-node-557fd3750d680c5b05b44cdc5d3ebf4972aeeeddd18c79744d805e13eaa3ec45.log
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=15123, records=16, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=1807489 file=/var/log/containers/kube-proxy-ssrlq_kube-system_kube-proxy-6d86c3689c16d8c7a9bdff2e42982f8f9bed657303457593fcf5ae58ab581b4e.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1807489 watch_fd=18 name=/var/log/containers/kube-proxy-ssrlq_kube-system_kube-proxy-6d86c3689c16d8c7a9bdff2e42982f8f9bed657303457593fcf5ae58ab581b4e.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 3.0K
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=2048, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 536b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=3837, records=4, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 813b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1794, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 282b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1794, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 282b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1792, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 280b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1794, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 282b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1795, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 283b
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853 file=/var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log promote to TAIL_EVENT
[2023/11/24 21:21:32] [ info] [input:tail:tail.0] inotify_fs_add(): inode=4128853 watch_fd=19 name=/var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] [static files] processed 0b, done
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=4607, records=5, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=2915, records=3, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=917, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=2845, records=3, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1927, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1928, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1929, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=1930, records=2, input=tail.0
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=918, records=1, input=tail.0
[2023/11/24 21:21:32] [debug] [input:tail:tail.0] inode=4128853, /var/log/containers/fluent-bit-demo-r9bj5_default_fluent-bit-52c68d0654561b70061e6467316cb7d418bdd3674cfb29741bcebd732303ff3f.log, events: IN_MODIFY
[2023/11/24 21:21:32] [debug] [input chunk] update output instances with new chunk size diff=919, records=1, input=tail.0
[2023/11/24 21:21:32] [ warn] [input] tail.0 paused (mem buf overlimit)
[2023/11/24 21:21:32] [ info] [input] pausing tail.0
[2023/11/24 21:21:33] [debug] [task] created task=0x7f3659a3ac40 id=0 OK
[2023/11/24 21:21:33] [debug] [task] created task=0x7f3659a3ad80 id=1 OK
[2023/11/24 21:21:33] [debug] [task] created task=0x7f3659a3aec0 id=2 OK
[2023/11/24 21:21:33] [trace] [output:azure_kusto:azure_kusto.0 at /src/fluent-bit/plugins/out_azure_kusto/azure_kusto.c:374] flushing bytes 33104
[2023/11/24 21:21:33] [ info] [output:azure_kusto:azure_kusto.0] loading kusto ingestion resourcs
[2023/11/24 21:21:33] [trace] [output:azure_kusto:azure_kusto.0 at /src/fluent-bit/plugins/out_azure_kusto/azure_kusto.c:374] flushing bytes 958237

• Steps to reproduce the problem:

Expected behavior

Screenshots

Your Environment

• Version used:
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?):
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context

[tanmaya-panda1]

complete-log.txt

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days. Maintainers can add the exempt-stale label.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.