fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Published 20 hours ago • verified publisher icon fluent

out_blob: consider auto_create_container

Open uristernik opened this issue 5 months ago • 3 comments

Consider auto_create_container when ensuring container exist

  1. Today we are not considering this setting
  2. Changing the behaviour that setting auto_create_container to false should return FLB_TRUE. If auto_create_container is false return FLB_FALSE, fluent-bit will never flush data to storage account, it will retry and eventually fail.

Fixes #9459

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [N/A] Example configuration file for the change
  • [N/A] Debug log output from testing the change No need for debug log output, attaching info log output

With auto_create_container on:

Fluent Bit v3.2.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/10/04 08:22:02] [ info] [fluent bit] version=3.2.0, commit=88f44c6c80, pid=1
[2024/10/04 08:22:02] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/10/04 08:22:02] [ info] [cmetrics] version=0.9.6
[2024/10/04 08:22:02] [ info] [ctraces ] version=0.5.6
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] initializing
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] multiline core started
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0
[2024/10/04 08:22:02] [ info] [filter:kubernetes:kubernetes.1] https=1 host=10.96.0.1 port=443
[2024/10/04 08:22:02] [ info] [filter:kubernetes:kubernetes.1]  token updated
[2024/10/04 08:22:02] [ info] [filter:kubernetes:kubernetes.1] local POD info OK
[2024/10/04 08:22:02] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...
[2024/10/04 08:22:02] [ info] [filter:kubernetes:kubernetes.1] connectivity OK
[2024/10/04 08:22:02] [ info] [output:azure_blob:azure_blob.0] account_name=xxx, container_name=logs, blob_type=blockblob, emulator_mode=no, endpoint=https://xxx/, auth_type=sas
[2024/10/04 08:22:02] [ info] [output:azure_blob:azure_blob.0] initializing worker
[2024/10/04 08:22:02] [ info] [output:azure_blob:azure_blob.0] worker #0 started
[2024/10/04 08:22:02] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/10/04 08:22:02] [ info] [sp] stream processor started
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361836 watch_fd=1 name=/var/log/containers/coredns-5dd5756b68-zxntv_kube-system_coredns-1151680f0cfdb669fb0aa0b67002b3e7cebde20951de9708cdefec126fb3622b.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1352059 watch_fd=2 name=/var/log/containers/coredns-5dd5756b68-zxntv_kube-system_coredns-61f36c9a9001fe1445db694e2b3bfbd5d29f29c311f117f7fca185769df06b2c.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361296 watch_fd=3 name=/var/log/containers/etcd-minikube_kube-system_etcd-94227cefaff3b3b2d3fba53eb691ae2c8e835b46c0dd91dbb6fdc764ff92b34a.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351393 watch_fd=4 name=/var/log/containers/etcd-minikube_kube-system_etcd-b820386d0619804d710ac9ca4d9511ab20f4cce008a7c6f0fbd5fe282453f6c0.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1180622 watch_fd=5 name=/var/log/containers/fluent-bit-d8kcc_default_fluent-bit-5efb3c8e4e9a655178c845674ee1e975593b44c1d35300bf7fd29b0d6f8684cd.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361715 watch_fd=6 name=/var/log/containers/kindnet-gdhj4_kube-system_kindnet-cni-96c5cef5c116d8723302b4c7b78251834315280ee2c5feb42e87fa0a54b70c70.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1352078 watch_fd=7 name=/var/log/containers/kindnet-gdhj4_kube-system_kindnet-cni-afcf957e18dad68983bcd576a11addc2616571923a3fa2fd2148e0efdcfe91e7.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351390 watch_fd=8 name=/var/log/containers/kube-apiserver-minikube_kube-system_kube-apiserver-73ef43bb9d99f99928e80273731975e6da9b10cc725d7484270e597f2d61ebcb.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361289 watch_fd=9 name=/var/log/containers/kube-apiserver-minikube_kube-system_kube-apiserver-fd0c9e5f665024632fe49cb12b5f17c9619b053cafdc6cc249091ed6dc2916d2.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351389 watch_fd=10 name=/var/log/containers/kube-controller-manager-minikube_kube-system_kube-controller-manager-ac5d19593b98a1d539cc5643c947fe0cd80279fac29237d1f79611e288480e8a.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361297 watch_fd=11 name=/var/log/containers/kube-controller-manager-minikube_kube-system_kube-controller-manager-e686474dfc80364029317aa43140e4dc5b47cad951aa472b1d1faff95a1e29e3.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361690 watch_fd=12 name=/var/log/containers/kube-proxy-7bmp5_kube-system_kube-proxy-4659c90c8afbc32e6494399f10ba19f156fa9cd5187a2eabf4ec8a9d877bfe57.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351834 watch_fd=13 name=/var/log/containers/kube-proxy-7bmp5_kube-system_kube-proxy-948535a87eef9ef20735cd86b9d025924f0a52455a4fa7d7e0da4a93ee811020.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361294 watch_fd=14 name=/var/log/containers/kube-scheduler-minikube_kube-system_kube-scheduler-357922c70e5c6ed04fc9bd7817a9aae39d076815838f3016c0897549bf11bf12.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351395 watch_fd=15 name=/var/log/containers/kube-scheduler-minikube_kube-system_kube-scheduler-b746b05b59757cac516f7a8e9fc6f3d42f867ad6884b96e0677cc445e36f381a.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361846 watch_fd=16 name=/var/log/containers/metrics-server-7c66d45ddc-w6lmf_kube-system_metrics-server-27ea1e3df003a11e68a56473622dfbde505b9e692eeeb0e6e6a0fc1ba8d0642d.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1357830 watch_fd=17 name=/var/log/containers/metrics-server-7c66d45ddc-w6lmf_kube-system_metrics-server-300fbb3cd38c48fba82effcd2ed5327f9ea14e18b0c992776d31694854cd64f6.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361693 watch_fd=18 name=/var/log/containers/storage-provisioner_kube-system_storage-provisioner-6f2fe8443b054cb954b588121eef231a8ab9177b073f52134f5b7cff161c0f12.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351900 watch_fd=19 name=/var/log/containers/storage-provisioner_kube-system_storage-provisioner-8a0e04f927948876fb14d3b4ed7e4fd3217335d808c80ae63dede52eb3d02f1c.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1180474 watch_fd=20 name=/var/log/containers/test_default_upload-1803814d839588840da38a65a5b6e974ffa65368146078d5ccef4ce16f835665.log
[2024/10/04 08:22:02] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1180641 watch_fd=21 name=/var/log/containers/fluent-bit-fb2bz_default_fluent-bit-2a04054b3448f7b1f83c2b71674ff262975f483b7eab473a2f009c4d4c8ea79b.log
[2024/10/04 08:22:03] [ info] [input:tail:tail.0] inotify_fs_remove(): inode=1180622 watch_fd=5
[2024/10/04 08:22:12] [error] [output:azure_blob:azure_blob.0] failed getting container 'logs', access denied
[2024/10/04 08:22:12] [error] [output:azure_blob:azure_blob.0] cannot ensure container 'logs' exists
[2024/10/04 08:22:12] [ warn] [engine] failed to flush chunk '1-1728030122.304633888.flb', retry in 10 seconds: task_id=1, input=tail.0 > output=azure_blob.0 (out_id=0)
[2024/10/04 08:22:12] [error] [output:azure_blob:azure_blob.0] failed getting container 'logs', access denied
[2024/10/04 08:22:12] [error] [output:azure_blob:azure_blob.0] cannot ensure container 'logs' exists
[2024/10/04 08:22:12] [ warn] [engine] failed to flush chunk '1-1728030125.262367626.flb', retry in 10 seconds: task_id=3, input=tail.0 > output=azure_blob.0 (out_id=0)
[2024/10/04 08:22:12] [error] [output:azure_blob:azure_blob.0] failed getting container 'logs', access denied
[2024/10/04 08:22:12] [error] [output:azure_blob:azure_blob.0] cannot ensure container 'logs' exists
[2024/10/04 08:22:12] [ warn] [engine] failed to flush chunk '1-1728030123.339554250.flb', retry in 9 seconds: task_id=2, input=tail.0 > output=azure_blob.0 (out_id=0)
[2024/10/04 08:22:12] [error] [output:azure_blob:azure_blob.0] failed getting container 'logs', access denied
[2024/10/04 08:22:12] [ warn] [engine] failed to flush chunk '1-1728030122.261692013.flb', retry in 6 seconds: task_id=0, input=tail.0 > output=azure_blob.0 (out_id=0)
[2024/10/04 08:22:12] [error] [output:azure_blob:azure_blob.0] cannot ensure container 'logs' exists

With auto_create_container off:

Fluent Bit v3.2.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____ 
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /  
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2024/10/04 08:28:48] [ info] [fluent bit] version=3.2.0, commit=88f44c6c80, pid=1
[2024/10/04 08:28:48] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/10/04 08:28:48] [ info] [cmetrics] version=0.9.6
[2024/10/04 08:28:48] [ info] [ctraces ] version=0.5.6
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] initializing
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] storage_strategy='memory' (memory only)
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] multiline core started
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] db: delete unmonitored stale inodes from the database: count=0
[2024/10/04 08:28:48] [ info] [filter:kubernetes:kubernetes.1] https=1 host=10.96.0.1 port=443
[2024/10/04 08:28:48] [ info] [filter:kubernetes:kubernetes.1]  token updated
[2024/10/04 08:28:48] [ info] [filter:kubernetes:kubernetes.1] local POD info OK
[2024/10/04 08:28:48] [ info] [filter:kubernetes:kubernetes.1] testing connectivity with API server...
[2024/10/04 08:28:48] [ info] [filter:kubernetes:kubernetes.1] connectivity OK
[2024/10/04 08:28:48] [ info] [output:azure_blob:azure_blob.0] account_name=xxx, container_name=logs, blob_type=blockblob, emulator_mode=no, endpoint=https://xxx/, auth_type=sas
[2024/10/04 08:28:48] [ info] [output:azure_blob:azure_blob.0] initializing worker
[2024/10/04 08:28:48] [ info] [output:azure_blob:azure_blob.0] worker #0 started
[2024/10/04 08:28:48] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020
[2024/10/04 08:28:48] [ info] [sp] stream processor started
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1180747 watch_fd=1 name=/var/log/containers/coredns-5d756df69c-8l9w9_kube-system_coredns-a379d17d983271d154da6d29351b48ffcb394d979f848faeac4ed0192a4ff2b3.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1180648 watch_fd=2 name=/var/log/containers/coredns-5d756df69c-zstzm_kube-system_coredns-1bf55c770e6a7080452d77109f8a64c16bfbde112ee31da3f7c0db7bb6750c68.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361296 watch_fd=3 name=/var/log/containers/etcd-minikube_kube-system_etcd-94227cefaff3b3b2d3fba53eb691ae2c8e835b46c0dd91dbb6fdc764ff92b34a.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351393 watch_fd=4 name=/var/log/containers/etcd-minikube_kube-system_etcd-b820386d0619804d710ac9ca4d9511ab20f4cce008a7c6f0fbd5fe282453f6c0.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361715 watch_fd=5 name=/var/log/containers/kindnet-gdhj4_kube-system_kindnet-cni-96c5cef5c116d8723302b4c7b78251834315280ee2c5feb42e87fa0a54b70c70.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1352078 watch_fd=6 name=/var/log/containers/kindnet-gdhj4_kube-system_kindnet-cni-afcf957e18dad68983bcd576a11addc2616571923a3fa2fd2148e0efdcfe91e7.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351390 watch_fd=7 name=/var/log/containers/kube-apiserver-minikube_kube-system_kube-apiserver-73ef43bb9d99f99928e80273731975e6da9b10cc725d7484270e597f2d61ebcb.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361289 watch_fd=8 name=/var/log/containers/kube-apiserver-minikube_kube-system_kube-apiserver-fd0c9e5f665024632fe49cb12b5f17c9619b053cafdc6cc249091ed6dc2916d2.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351389 watch_fd=9 name=/var/log/containers/kube-controller-manager-minikube_kube-system_kube-controller-manager-ac5d19593b98a1d539cc5643c947fe0cd80279fac29237d1f79611e288480e8a.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361297 watch_fd=10 name=/var/log/containers/kube-controller-manager-minikube_kube-system_kube-controller-manager-e686474dfc80364029317aa43140e4dc5b47cad951aa472b1d1faff95a1e29e3.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361690 watch_fd=11 name=/var/log/containers/kube-proxy-7bmp5_kube-system_kube-proxy-4659c90c8afbc32e6494399f10ba19f156fa9cd5187a2eabf4ec8a9d877bfe57.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351834 watch_fd=12 name=/var/log/containers/kube-proxy-7bmp5_kube-system_kube-proxy-948535a87eef9ef20735cd86b9d025924f0a52455a4fa7d7e0da4a93ee811020.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361294 watch_fd=13 name=/var/log/containers/kube-scheduler-minikube_kube-system_kube-scheduler-357922c70e5c6ed04fc9bd7817a9aae39d076815838f3016c0897549bf11bf12.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351395 watch_fd=14 name=/var/log/containers/kube-scheduler-minikube_kube-system_kube-scheduler-b746b05b59757cac516f7a8e9fc6f3d42f867ad6884b96e0677cc445e36f381a.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361846 watch_fd=15 name=/var/log/containers/metrics-server-7c66d45ddc-w6lmf_kube-system_metrics-server-27ea1e3df003a11e68a56473622dfbde505b9e692eeeb0e6e6a0fc1ba8d0642d.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1357830 watch_fd=16 name=/var/log/containers/metrics-server-7c66d45ddc-w6lmf_kube-system_metrics-server-300fbb3cd38c48fba82effcd2ed5327f9ea14e18b0c992776d31694854cd64f6.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1361693 watch_fd=17 name=/var/log/containers/storage-provisioner_kube-system_storage-provisioner-6f2fe8443b054cb954b588121eef231a8ab9177b073f52134f5b7cff161c0f12.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1351900 watch_fd=18 name=/var/log/containers/storage-provisioner_kube-system_storage-provisioner-8a0e04f927948876fb14d3b4ed7e4fd3217335d808c80ae63dede52eb3d02f1c.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1180474 watch_fd=19 name=/var/log/containers/test_default_upload-1803814d839588840da38a65a5b6e974ffa65368146078d5ccef4ce16f835665.log
[2024/10/04 08:28:48] [ info] [input:tail:tail.0] inotify_fs_add(): inode=1180580 watch_fd=20 name=/var/log/containers/fluent-bit-8krf8_default_fluent-bit-2eb1d8867ac496f86b811eee21c59d2a6b4d1fd43c8930323bd67a3cf60c0b68.log
[2024/10/04 08:28:58] [ info] [output:azure_blob:azure_blob.0] auto_create_container is disabled, assuming container 'logs' already exists
[2024/10/04 08:28:58] [ info] [output:azure_blob:azure_blob.0] auto_create_container is disabled, assuming container 'logs' already exists
[2024/10/04 08:28:58] [ info] [output:azure_blob:azure_blob.0] content uploaded successfully: file=kube.var.log.containers.kindnet-gdhj4_kube-system_kindnet-cni-96c5cef5c116d8723302b4c7b78251834315280ee2c5feb42e87fa0a54b70c70.log.1728030538236
[2024/10/04 08:28:58] [ info] [output:azure_blob:azure_blob.0] blob id ZmxiLTE3MjgwMzA1MzguMjM2OC5pZA== committed successfully
[2024/10/04 08:28:59] [ info] [output:azure_blob:azure_blob.0] content uploaded successfully: file=kube.var.log.containers.fluent-bit-8krf8_default_fluent-bit-2eb1d8867ac496f86b811eee21c59d2a6b4d1fd43c8930323bd67a3cf60c0b68.log.1728030538233
  • [ ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [N/A] Documentation required for this feature

Backporting

  • [ ] Backport to latest stable release. Will do after this one is merged

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

uristernik avatar Oct 02 '24 10:10 uristernik