fluent-bit
fluent-bit copied to clipboard
Published
20 hours ago •
fluent
fluent
High memory usage
Bug Report
Describe the bug
Very few log files, but really high memory usage
fluent-bit: fluent-bit:1.9.0
NAME CPU(cores) MEMORY(bytes)
alertmanager-prometheus-kube-prometheus-alertmanager-0 2m 47Mi
fluent-bit-6rglj 37m 1102Mi
fluent-bit-ckmqw 2m 9Mi
fluent-bit-hswrv 2m 11Mi
fluent-bit-kfp7d 9m 81Mi
fluent-bit-mvqq5 2m 9Mi
fluent-bit-ptvhr 90m 2457Mi
fluent-bit-rb2jj 58m 1258Mi
fluent-bit-sf8n2 24m 111Mi
fluent-bit-sk7q5 55m 2283Mi
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scanning path /var/log/containers/*.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/apisix-etcd-2_ingress-apisix_etcd-4f703a889ed78911e4d1ebfc7fa853fd8178272082a09a22a44113f8be54f623.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/apisix-etcd-2_ingress-apisix_etcd-6801c3613526d5877576908b7ef3c13e029407dbe8820f620b038ca911ad059c.log, inode 13238347
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_calico-node-d616da859c552d8f277514d30842d18ff6163a753b28606730bcb2b7c351e85a.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_calico-node-dbff93e5664fac14dfce6bc0c11d3b309d0da5964a6cd0ad219fc5e44d10b102.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_flexvol-driver-98ad28b4f48e62957fda5542b585cd0a1e8229e7438c66a7126cd51e1c880ca4.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_install-cni-126861a53de82eb2d82bd4e04aea408419be4929d346c4da4b60443ad7cb6bbd.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_kube-flannel-0dfa1b98332458518f032d693b727dc505e693922540cf122643dcd2863cbc4b.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/canal-8rsg8_kube-system_kube-flannel-6ee75bd916181d22e06dc108b1780e39f0f58331521ed0e462b658af1755ec87.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/cattle-node-agent-9lr9k_cattle-system_agent-3736ccef4a78bf19ee94392fc4facc0e9f2818971774f327b8fd1998f0e0492b.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/cattle-node-agent-9lr9k_cattle-system_agent-f7c334919e4b0574761ee587eb960ff789e096a945c61e4c8131cf5792592ffe.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-qx6ns_consul_client-acl-init-f613e962f1e7d6b620afe2e6618b593a4464c92ad992fcb0d848498eae343e5e.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-qx6ns_consul_consul-933b73dd543a261b48655e5cf9c82452ada82ff2b43f566d993d01638ec59e5b.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/consul-consul-qx6ns_consul_consul-f8a71c8b685b9ae6beafd9980c84ba6d024abfd7708484270d28a9a562522398.log, inode 13238319
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-qx6ns_consul_linkerd-init-ecf90ca5e179038aeee43f280be6c66a116eef0e8f3e5a99b56ce6852cc71a18.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-qx6ns_consul_linkerd-proxy-32a76c1dd299b7e38b442198e758c56ae33fe6627eac8c9cd8b7441362687bc8.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/consul-consul-qx6ns_consul_linkerd-proxy-afe9a713ed29b5c580ed2ef36520c503e99a96d744ad746f55062cafae649960.log, inode 13238325
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-server-1_consul_consul-06b028e5d670acc12c9168ac5b1f50bbb68c0bae234ddad2f6f06bc0524846f5.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/consul-consul-server-1_consul_consul-407efd1759329967ade623d7ed9b352b21c330832b9806f94a6c390eb3fb2ef5.log, inode 22675569
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-server-1_consul_linkerd-init-8f7162f2731db471a4a91eda12ead3c4493b0f5282a3716e439db2e89e2d1f65.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/consul-consul-server-1_consul_linkerd-proxy-480bfa2638208a07a9e20232a9ab4f54f760d7eee5d07074266331bfd9133ec1.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/consul-consul-server-1_consul_linkerd-proxy-495fa5a19e35cc8dede94c85122d64217afa6b671fb4d9da87b87b762244f391.log, inode 13238312
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/csi-nodeplugin-fluid-fsb8z_fluid-system_node-driver-registrar-47868b7c08f2f3f54c6066fcedeee2d5a1de64244c00ba3eed1bdc8d55d6e1a8.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/csi-nodeplugin-fluid-fsb8z_fluid-system_plugins-e1db623a637ccf54ba103f1f36cb932b48d72e3849de49c4e5d6ae1793d19120.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/fluent-bit-ptvhr_logging_fluent-bit-34c01c77f93933596adcc4d857b8eed0eb15dbb749ca72a640fc5d33c0e0abce.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_grafana-3729b494dc39cf6bd342a872db7bb2974d7af30b998c3628a6adc8ee8afccd84.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_grafana-d58c2d63034f076a6d834753fa08a7871f396094619d4190b718bdf7f27ab6ab.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_linkerd-init-c890bc831d786b156e70618b64afe637c0455d04d35a1f072f04b76d1408c7ab.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_linkerd-proxy-543022e08d8947314a3f7b29fd99d8f2c5ad15fb282c7a3dde09e3ed02952931.log, inode 13238287
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/grafana-7c4bc49848-9d8ng_linkerd-viz_linkerd-proxy-846b4b566a08a19ee121df6e982a4ea1a4dee4b34d45c7d62f1cb76f454a2788.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/openaios-airbyte-bootloader_airbyte_airbyte-bootloader-container-eb9129259f2c44445d89b83fa048eab5c4ee25726917d2af68fba2180b024c3f.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/openaios-iam-5f66bcf696-6drc2_nlp-iam_openaios-iam-2ecefb36794c4644bb054abe91d0bbe35764fc4c4c32fbc0e4167d8a2107a9bb.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/openaios-iam-7757897c5b-fxx4t_openaios-iam_openaios-iam-1c08115d3da4185bd74faf4d0f099e423d41b84d50c0b7afcaeba581d3143b24.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/openaios-iam-7757897c5b-fxx4t_openaios-iam_openaios-iam-d9b241c75b49f46f51d70870c9c870853184b9eb56c10c857c7dd8218c7c1c30.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/prometheus-node-exporter-q9zf2_logging_node-exporter-835a7511a37f26456c89820135b49c333a42e2ba0f3d4312458507681fcc4c01.log
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/tap-8589bb5899-72btb_linkerd-viz_linkerd-init-38f0478d3995764e08fba1e47335faf47ad119f4fa62f76c643f556dc7ba835d.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/tap-8589bb5899-72btb_linkerd-viz_linkerd-proxy-69f1b93859330a30decd017c21ee97c414fa419f1fec3abcf0f9f22d504d0d3b.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/tap-8589bb5899-72btb_linkerd-viz_linkerd-proxy-fbb33fead9dbcd15d1de5049dbd44a4492fbeb171fb8cf9b953a0b29a1696170.log, inode 13238278
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/tap-8589bb5899-72btb_linkerd-viz_tap-e5637643561fb641be3dfe8b278f25944f569e0366fae6ae1ac68f0a86bf57db.log, inode 22807104
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/tap-injector-65d747bd67-lbl72_linkerd-viz_tap-injector-14ace39bf19b1d576987692c23460f3efe94352e4ed4b0d210a05081248674a5.log, inode 13238432
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/tap-injector-65d747bd67-lbl72_linkerd-viz_tap-injector-9e2eba69e03c424c6c93a1e731520e162fbd87a179e5913bc4f043bb9c444a24.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] scan_blog add(): dismissed: /var/log/containers/vote-bot-6cd4c65ccd-hnzsn_emojivoto_vote-bot-1b5b9172813a7c311c4cdfff7f0c684a61c34bd7845e63be3dae8b7b63a2fcf9.log, inode 13238285
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] excluded=/var/log/containers/vote-bot-6cd4c65ccd-hnzsn_emojivoto_vote-bot-6092ae56444d9869f80cb9965f1f55dd6fe2b49eeb82d431d274d712d0c0666a.log (ignore_older)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] 0 new files found on path '/var/log/containers/*.log'
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] inode=13238312 events: IN_MODIFY
[2022/09/21 17:37:40] [debug] [input chunk] update output instances with new chunk size diff=1489
[2022/09/21 17:37:40] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:40] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:40] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:40] [debug] [upstream] KA connection #34 to loki:3100 has been assigned (recycled)
[2022/09/21 17:37:40] [debug] [http_client] not using http_proxy for header
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY
[2022/09/21 17:37:40] [debug] [input chunk] update output instances with new chunk size diff=1394
[2022/09/21 17:37:40] [debug] [output:loki:loki.0] loki:3100, HTTP status=204
[2022/09/21 17:37:40] [debug] [upstream] KA connection #34 to loki:3100 is now available
[2022/09/21 17:37:40] [debug] [out flush] cb_destroy coro_id=2618
[2022/09/21 17:37:40] [debug] [task] destroy task=0x7fcb8fc54cb0 (task_id=516)
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY
[2022/09/21 17:37:40] [debug] [input chunk] update output instances with new chunk size diff=1307
[2022/09/21 17:37:40] [debug] [input:tail:tail.0] inode=13238325 events: IN_MODIFY
[2022/09/21 17:37:40] [debug] [input chunk] update output instances with new chunk size diff=1451
[2022/09/21 17:37:41] [debug] [input:tail:tail.0] inode=13238285 events: IN_MODIFY
[2022/09/21 17:37:41] [debug] [input chunk] update output instances with new chunk size diff=749
[2022/09/21 17:37:41] [debug] [task] created task=0x7fcb8fc54cb0 id=516 OK
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:41] [debug] [upstream] KA connection #34 to loki:3100 has been assigned (recycled)
[2022/09/21 17:37:41] [debug] [http_client] not using http_proxy for header
[2022/09/21 17:37:41] [debug] [output:loki:loki.0] loki:3100, HTTP status=204
[2022/09/21 17:37:41] [debug] [upstream] KA connection #34 to loki:3100 is now available
[2022/09/21 17:37:41] [debug] [out flush] cb_destroy coro_id=2619
[2022/09/21 17:37:41] [debug] [task] destroy task=0x7fcb92c43ce0 (task_id=50)
[2022/09/21 17:37:41] [debug] [input:tail:tail.0] inode=13238325 events: IN_MODIFY
[2022/09/21 17:37:41] [debug] [input chunk] update output instances with new chunk size diff=1275
[2022/09/21 17:37:41] [debug] [input chunk] update output instances with new chunk size diff=1262
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=1307
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238285 events: IN_MODIFY
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=748
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238347 events: IN_MODIFY
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=904
[2022/09/21 17:37:42] [debug] [task] created task=0x7fcb92c43ce0 id=50 OK
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=1394
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:42] [debug] [upstream] KA connection #34 to loki:3100 has been assigned (recycled)
[2022/09/21 17:37:42] [debug] [http_client] not using http_proxy for header
[2022/09/21 17:37:42] [debug] [output:loki:loki.0] loki:3100, HTTP status=204
[2022/09/21 17:37:42] [debug] [upstream] KA connection #34 to loki:3100 is now available
[2022/09/21 17:37:42] [debug] [out flush] cb_destroy coro_id=2620
[2022/09/21 17:37:42] [debug] [task] destroy task=0x7fcb8fc54c40 (task_id=517)
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238319 events: IN_MODIFY
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=1180
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=1250
[2022/09/21 17:37:42] [debug] [input:tail:tail.0] inode=13238347 events: IN_MODIFY
[2022/09/21 17:37:42] [debug] [input chunk] update output instances with new chunk size diff=903
[2022/09/21 17:37:43] [debug] [input:tail:tail.0] inode=13238285 events: IN_MODIFY
[2022/09/21 17:37:43] [debug] [input chunk] update output instances with new chunk size diff=748
[2022/09/21 17:37:43] [debug] [task] created task=0x7fcb8fc54c40 id=517 OK
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] could not translate record accessor
[2022/09/21 17:37:43] [debug] [upstream] KA connection #34 to loki:3100 has been assigned (recycled)
[2022/09/21 17:37:43] [debug] [http_client] not using http_proxy for header
[2022/09/21 17:37:43] [debug] [output:loki:loki.0] loki:3100, HTTP status=204
[2022/09/21 17:37:43] [debug] [upstream] KA connection #34 to loki:3100 is now available
[2022/09/21 17:37:43] [debug] [out flush] cb_destroy coro_id=2621
[2022/09/21 17:37:43] [debug] [task] destroy task=0x7fcb8fc54bd0 (task_id=518)
[2022/09/21 17:37:43] [debug] [input:tail:tail.0] inode=13238278 events: IN_MODIFY
[2022/09/21 17:37:43] [debug] [input chunk] update output instances with new chunk size diff=1307
What's your config? Please follow the issue template as it really helps to capture all the relevant information.
Is this with fresh pods, i.e. no persisted data that has not been sent? It looks like some failures in the Loki output, are logs being sent for this?
thank you @patrick-stephens
my fluent-bit config:
apiVersion: v1
data:
custom_parsers.conf: |
[PARSER]
Name docker_no_time
Format json
Time_Keep Off
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L
fluent-bit.conf: |
[SERVICE]
Daemon Off
Flush 1
Log_Level debug
Parsers_File parsers.conf
Parsers_File custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On
[INPUT]
Name tail
Path /var/log/containers/*.log
multiline.parser docker, cri
Tag kube.*
Ignore_Older 1h
Exclude_Path *_kube-system_*.log,*_logging_*.log
Refresh_Interval 5
Skip_Long_Lines On
DB /var/log/flb_kube.db
Mem_Buf_Limit 1024MB
[FILTER]
Name kubernetes
Match kube.*
Buffer_Size 512KB
Merge_Log On
Keep_Log Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[OUTPUT]
Name loki
Match kube.*
Host loki
Port 3100
tenant_id ""
Labels agent=fluent-bit,pod=$kubernetes['pod_name'],namespace=$kubernetes['namespace_name'],name=$kubernetes['labels']['app.kubernetes.io/name'],version=$kubernetes['labels']['app.kubernetes.io/version'],node=$kubernetes['host'],container=$kubernetes['container_name'],instance=$kubernetes['labels']['app.kubernetes.io/instance']
Label_Keys $stream
auto_kubernetes_labels off
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: fluent-bit
meta.helm.sh/release-namespace: logging
creationTimestamp: "2022-09-19T02:22:59Z"
labels:
app.kubernetes.io/instance: fluent-bit
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: fluent-bit
app.kubernetes.io/version: 1.9.8
helm.sh/chart: fluent-bit-0.20.8
name: fluent-bit
namespace: logging
resourceVersion: "126984510"
uid: 1c987657-cf5b-4e27-9031-7000e3a07d2a
loki log info:
level=info ts=2022-09-21T13:08:29.89651657Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:08:29.896537122Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:09:29.801949192Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:09:29.802035566Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:09:29.802070393Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:09:29.802097406Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:09:29.80212348Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:09:29.802145177Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:09:29.802174013Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:09:29.896280837Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:09:29.896416004Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:09:29.896450777Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:09:29.896536488Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:09:29.896559498Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:09:45.000821372Z caller=checkpoint.go:502 msg="atomic checkpoint finished" old=/data/loki/wal/checkpoint.000124.tmp new=/data/loki/wal/checkpoint.000124
level=info ts=2022-09-21T13:09:45.011993148Z caller=checkpoint.go:573 msg="checkpoint done" time=4m15.014466686s
level=info ts=2022-09-21T13:10:29.80203894Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:10:29.802118817Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:10:29.802139647Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:10:29.802156695Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:10:29.802175513Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:10:29.802187786Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:10:29.802201199Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:10:29.89680063Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:10:29.896832154Z caller=table_manager.go:213 msg="syncing tables"
ts=2022-09-21T13:10:29.896958618Z caller=spanlogger.go:80 level=info msg="building index list cache"
level=info ts=2022-09-21T13:10:29.896972165Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:10:29.897000801Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:10:29.897094435Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:10:29.897134342Z caller=table.go:335 msg="finished handing over table index_19256"
ts=2022-09-21T13:10:29.897255149Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=251.191µs
level=info ts=2022-09-21T13:10:29.897342722Z caller=table_manager.go:252 msg="query readiness setup completed" duration=2.37µs distinct_users_len=0
level=info ts=2022-09-21T13:10:29.99703226Z caller=checkpoint.go:615 msg="starting checkpoint"
level=info ts=2022-09-21T13:10:29.997334651Z caller=checkpoint.go:340 msg="attempting checkpoint for" dir=/data/loki/wal/checkpoint.000125
ts=2022-09-21T13:10:36.099466264Z caller=spanlogger.go:80 level=info msg="building index list cache"
ts=2022-09-21T13:10:36.099744711Z caller=spanlogger.go:80 level=info msg="index list cache built" duration=172.947µs
level=info ts=2022-09-21T13:10:36.099819808Z caller=compactor.go:552 msg="compacting table" table-name=index_19255
level=info ts=2022-09-21T13:10:36.099955089Z caller=table.go:138 table-name=index_19255 msg="listed files" count=1
level=info ts=2022-09-21T13:10:36.100043789Z caller=compactor.go:557 msg="finished compacting table" table-name=index_19255
level=info ts=2022-09-21T13:10:36.100070887Z caller=compactor.go:552 msg="compacting table" table-name=index_19256
level=info ts=2022-09-21T13:10:36.100145732Z caller=table.go:138 table-name=index_19256 msg="listed files" count=2
level=info ts=2022-09-21T13:10:36.100167532Z caller=table.go:297 table-name=index_19256 msg="starting compaction of dbs"
level=info ts=2022-09-21T13:10:36.100210147Z caller=table.go:307 table-name=index_19256 msg="using compactor-1663764636.gz as seed file"
level=info ts=2022-09-21T13:10:36.109680003Z caller=util.go:116 table-name=index_19256 file-name=compactor-1663764636.gz msg="downloaded file" total_time=9.45011ms
level=info ts=2022-09-21T13:10:36.110553237Z caller=util.go:116 table-name=index_19256 file-name=loki-0-1663728029801110696-1663764300.gz msg="downloaded file" total_time=672.745µs
level=info ts=2022-09-21T13:10:36.124081021Z caller=util.go:136 msg="compressing the file" src=/data/loki/boltdb-shipper-compactor/index_19256/compactor-1663764636.gz dest=/data/loki/boltdb-shipper-compactor/index_19256/compactor-1663764636.gz.gz
level=info ts=2022-09-21T13:10:36.296783281Z caller=index_set.go:281 table-name=index_19256 msg="removing source db files from storage" count=2
level=info ts=2022-09-21T13:10:36.297714292Z caller=compactor.go:557 msg="finished compacting table" table-name=index_19256
level=info ts=2022-09-21T13:11:29.801821335Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:11:29.801895922Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:11:29.801918072Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:11:29.801939308Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:11:29.801956512Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:11:29.801970722Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:11:29.801986375Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:11:29.896069463Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:11:29.896192401Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:11:29.896216892Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:11:29.896290039Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:11:29.896311959Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:12:29.801669404Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:12:29.801773208Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:12:29.801794888Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:12:29.801812861Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:12:29.801830821Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:12:29.801845261Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:12:29.801862381Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:12:29.896512019Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:12:29.896631606Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:12:29.89666187Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:12:29.89674563Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:12:29.896766387Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:13:29.801551793Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:13:29.801632587Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:13:29.801654037Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:13:29.801671857Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:13:29.801689057Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:13:29.801732051Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:13:29.801748684Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:13:29.896828604Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:13:29.896976898Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:13:29.897002893Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:13:29.897065941Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:13:29.897085933Z caller=table.go:335 msg="finished handing over table index_19256"
level=info ts=2022-09-21T13:14:29.801861489Z caller=table_manager.go:134 msg="uploading tables"
level=info ts=2022-09-21T13:14:29.801927982Z caller=index_set.go:86 msg="uploading table index_19256"
level=info ts=2022-09-21T13:14:29.8019487Z caller=index_set.go:107 msg="finished uploading table index_19256"
level=info ts=2022-09-21T13:14:29.801964681Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19256"
level=info ts=2022-09-21T13:14:29.801981835Z caller=index_set.go:86 msg="uploading table index_19255"
level=info ts=2022-09-21T13:14:29.801994078Z caller=index_set.go:107 msg="finished uploading table index_19255"
level=info ts=2022-09-21T13:14:29.802007436Z caller=index_set.go:185 msg="cleaning up unwanted indexes from table index_19255"
level=info ts=2022-09-21T13:14:29.89681816Z caller=table_manager.go:167 msg="handing over indexes to shipper"
level=info ts=2022-09-21T13:14:29.896928704Z caller=table.go:319 msg="handing over indexes to shipper index_19255"
level=info ts=2022-09-21T13:14:29.896952804Z caller=table.go:335 msg="finished handing over table index_19255"
level=info ts=2022-09-21T13:14:29.897007928Z caller=table.go:319 msg="handing over indexes to shipper index_19256"
level=info ts=2022-09-21T13:14:29.897030271Z caller=table.go:335 msg="finished handing over table index_19256"
Hi @patrick-stephens I have the same issue, I'm using Fluent-bit 1.9.0 and it is deployed in kubernetes cluster as deployment. There's no data flowing through fluent-bit at this time when I check its dump metrics, and the memory size in input buffer and tasks is low:
[2022/09/21 15:05:53] Fluent Bit Dump
===== Input ===== tail.0 (tail) │ ├─ status │ └─ overlimit : no │ ├─ mem size : 22.5K (23003 bytes) │ └─ mem limit : 4.8M (5000000 bytes) │ ├─ tasks │ ├─ total tasks : 0 │ ├─ new : 0 │ ├─ running : 0 │ └─ size : 0b (0 bytes) │ └─ chunks └─ total chunks : 2 ├─ up chunks : 2 ├─ down chunks: 0 └─ busy chunks: 0 ├─ size : 0b (0 bytes) └─ size err: 0
forward.1 (forward) │ ├─ status │ └─ overlimit : no │ ├─ mem size : 0b (0 bytes) │ └─ mem limit : 0b (0 bytes) │ ├─ tasks │ ├─ total tasks : 0 │ ├─ new : 0 │ ├─ running : 0 │ └─ size : 0b (0 bytes) │ └─ chunks └─ total chunks : 0 ├─ up chunks : 0 ├─ down chunks: 0 └─ busy chunks: 0 ├─ size : 0b (0 bytes) └─ size err: 0
===== Storage Layer ===== total chunks : 2 ├─ mem chunks : 2 └─ fs chunks : 0 ├─ up : 0 └─ down : 0
But its memory usage checking by 'kubectl top pod' is 62Mi. I wonder why the gap is so large...
$ kubectl top pod -A | grep fluent-bit pks-system fluent-bit-gbcfm 7m 62Mi
The fluent-bit configuration : fluent-bit.conf:
[SERVICE] Flush 1 Log_Level warning Daemon off Parsers_File parsers.conf HTTP_Server On HTTP_Listen 0.0.0.0 HTTP_Port 2020
input-kubernetes.conf:
[INPUT] Name tail Tag kube.* Path /var/log/containers/*.log DB /var/log/flb_kube.db Mem_Buf_Limit 5MB Skip_Long_Lines On Refresh_Interval 10 multiline.parser docker, cri`
outputs.conf:
[OUTPUT] Name splunk Match default Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match k8s-certmanager Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match k8s-dashboard Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match k8s-falco Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match k8s-gatekeeper Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match k8s-monitoring Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match k8s-trident Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match k8s-x509-cert-exporter Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match kube-node-lease Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match kube-public Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match kube-system Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match nsx-system Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name forward Match pca-uat1-accessdecision Host *** Port 24224
[OUTPUT] Name forward Match pca-uat2-accessdecision Host fluentbit-fluent-bit.pch-uat2-shared Port 24224
[OUTPUT] Name forward Match pca-uat3-accessdecision Host fluentbit-fluent-bit.pch-uat3-shared Port 24224
[OUTPUT] Name forward Match pcc-uat1-contactnote Host fluentbit-fluent-bit.pch-uat1-shared Port 24224
[OUTPUT] Name forward Match pcc-uat2-contactnote Host fluentbit-fluent-bit.pch-uat2-shared Port 24224
[OUTPUT] Name forward Match pcc-uat3-contactnote Host fluentbit-fluent-bit.pch-uat3-shared Port 24224
[OUTPUT] Name forward Match pcd-uat1-pcmdocmgmt Host fluentbit-fluent-bit.pch-uat1-shared Port 24224
[OUTPUT] Name forward Match pcm-uat1-pcmcore Host fluentbit-fluent-bit.pch-uat1-shared Port 24224
[OUTPUT] Name forward Match pcm-uat2-pcmcore Host fluentbit-fluent-bit.pch-uat2-shared Port 24224
[OUTPUT] Name forward Match pcm-uat3-pcmcore Host fluentbit-fluent-bit.pch-uat3-shared Port 24224
[OUTPUT] Name splunk Match pks-system Host srp15913lx.juliusbaer.com Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name splunk Match pks-system-host-monitoring Host *** Port 8088 Splunk_Token *** tls true tls.verify false
[OUTPUT] Name forward Match pmk-uat1-pcmtask Host fluentbit-fluent-bit.pch-uat1-shared Port 24224
[OUTPUT] Name forward Match pmk-uat2-pcmtask Host fluentbit-fluent-bit.pch-uat2-shared Port 24224
[OUTPUT] Name forward Match pmk-uat3-pcmtask Host fluentbit-fluent-bit.pch-uat3-shared Port 24224
[OUTPUT] Name splunk Match vmware-system-csi Host *** Port 8088 Splunk_Token *** tls true tls.verify false
I would also try the latest version 1.9.9 too to confirm it is still present.
This problem was solved by following @sflanker strategy
https://github.com/fluent/fluent-bit/issues/5868#issuecomment-1263282983
Hi @allenhaozi We have the same issue with splunk output plugin, can you confirm if it's the same issue?
hi @wangyuan0916 I followed this plan to upgrade the image and configuration, observed the whole afternoon, so far it has been solved https://github.com/fluent/fluent-bit/issues/5868#issuecomment-1263282983
$k top pods
NAME CPU(cores) MEMORY(bytes)
fluent-bit-2s4vf 4m 29Mi
fluent-bit-6ksnk 6m 26Mi
fluent-bit-7cf6q 3m 19Mi
fluent-bit-7p8b7 20m 85Mi
fluent-bit-fm5fk 13m 84Mi
fluent-bit-ggssv 5m 75Mi
fluent-bit-mfz47 7m 90Mi
fluent-bit-pxsgv 7m 127Mi
fluent-bit-tcfbb 4m 28Mi
loki-0 13m 289Mi