fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard

Published 20 hours ago • verified publisher icon fluent

out_cloudwatch_logs: support log_group_class option

Open PettitWesley opened this issue 1 year ago • 3 comments

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [ ] Example configuration file for the change
  • [ ] Debug log output from testing the change
  • [ ] Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [ ] Run local packaging test showing all targets (including any new ones) build.
  • [ ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [ ] Documentation required for this feature

Backporting

  • [ ] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

PettitWesley avatar May 17 '24 04:05 PettitWesley

147dda285fd4:bottlerocket-eks wppttt$ kubectl -n amazon-cloudwatch logs fluent-bit-fbf8584f8-vvsnl Fluent Bit v1.9.10

  • Copyright (C) 2015-2022 The Fluent Bit Authors
  • Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
  • https://fluentbit.io

[2024/05/17 03:47:44] [ info] [fluent bit] version=1.9.10, commit=7eb6b397eb, pid=1 [2024/05/17 03:47:44] [ info] [storage] version=1.4.0, type=memory+filesystem, sync=normal, checksum=disabled, max_chunks_up=128 [2024/05/17 03:47:44] [ info] [storage] backlog input plugin: storage_backlog.8 [2024/05/17 03:47:44] [ info] [cmetrics] version=0.3.7 [2024/05/17 03:47:44] [ info] [input:tail:tail.0] multiline core started [2024/05/17 03:47:44] [ info] [input:tail:tail.1] multiline core started [2024/05/17 03:47:44] [ info] [input:tail:tail.2] multiline core started [2024/05/17 03:47:44] [ info] [input:systemd:systemd.3] seek_cursor=s=fd652eba798a44cc9ecbea19cdd2ac3e;i=1af... OK [2024/05/17 03:47:44] [ info] [input:tail:tail.4] multiline core started [2024/05/17 03:47:44] [ info] [input:storage_backlog:storage_backlog.8] queue memory limit: 4.8M [2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] https=1 host=127.0.0.1 port=10250 [2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] token updated [2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] local POD info OK [2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] testing connectivity with Kubelet... [2024/05/17 03:47:44] [ info] [filter:kubernetes:kubernetes.0] connectivity OK [2024/05/17 03:47:44] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Configured log_group_class INFREQUENT_ACCESS will only apply to log groups created by Fluent Bit. Look for the Created log group info level message emitted when a group does not already exist and is created. [2024/05/17 03:47:44] [ warn] [output:cloudwatch_logs:cloudwatch_logs.1] Configured log_group_class INFREQUENT_ACCESS will only apply to log groups created by Fluent Bit. Look for the Created log group info level message emitted when a group does not already exist and is created. [2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] worker #0 started [2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] worker #0 started [2024/05/17 03:47:44] [ info] [output:cloudwatch_logs:cloudwatch_logs.2] worker #0 started [2024/05/17 03:47:44] [ info] [http_server] listen iface=0.0.0.0 tcp_port=2020 [2024/05/17 03:47:44] [ info] [sp] stream processor started [2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=42307 watch_fd=1 name=/var/log/containers/coredns-68c6b7b454-r6t8q_kube-system_coredns-1709ec5295314385997364635ee603e6d6a559f6cfb70bf3f3e2c431355833c5.log [2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=167812073 watch_fd=2 name=/var/log/containers/coredns-68c6b7b454-tc5mh_kube-system_coredns-2cd2d04c3f2bc1460ac0e86dbd1c3b3b3f8ba0828fc97cc122b491ce897b457e.log [2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=201708930 watch_fd=3 name=/var/log/containers/eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-init-809c5a47b01cca41b44dc18b54b81ab3f3753d1caae2c0c0ca1f6439b5fa4ae3.log [2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=218165991 watch_fd=1 name=/var/log/containers/aws-node-jhslt_kube-system_aws-eks-nodeagent-f083e3e56bc143254ca23011138731a632725afa821c9fc1f442f4e0901eef29.log [2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=16777403 watch_fd=2 name=/var/log/containers/aws-node-jhslt_kube-system_aws-node-7cd53aacd33b6193b69e8085093fe6876e3d0dcda16f4003813528a7c378059e.log [2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=117531748 watch_fd=3 name=/var/log/containers/aws-node-jhslt_kube-system_aws-vpc-cni-init-e5fe7f223d165f4b1c55d1bcc3b8e9f47aa1a475f45d8b3e83c2a1070e82f941.log [2024/05/17 03:47:44] [ info] [input:tail:tail.0] inotify_fs_add(): inode=185075710 watch_fd=4 name=/var/log/containers/eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log [2024/05/17 03:47:44] [ info] [input:tail:tail.1] inotify_fs_add(): inode=218429523 watch_fd=1 name=/var/log/containers/fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log [2024/05/17 03:47:44] [ info] [input:tail:tail.4] inotify_fs_add(): inode=100789792 watch_fd=4 name=/var/log/containers/kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it. [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it. [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it. [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane not found. Will attempt to create it. [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it. [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.' [2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group [2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events [2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.974275925.flb', retry in 10 seconds: task_id=4, input=tail.4 > output=cloudwatch_logs.1 (out_id=1) [2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.' [2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group [2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events [2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.948458095.flb', retry in 10 seconds: task_id=3, input=systemd.3 > output=cloudwatch_logs.1 (out_id=1) [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane with storage class INFREQUENT_ACCESS [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application with storage class INFREQUENT_ACCESS [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it. [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status. [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.containerd.service [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it. [2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status. [2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events [2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.940218327.flb', retry in 7 seconds: task_id=1, input=tail.1 > output=cloudwatch_logs.0 (out_id=0) [2024/05/17 03:47:49] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status. [2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events [2024/05/17 03:47:49] [ warn] [engine] failed to flush chunk '1-1715917664.929877801.flb', retry in 6 seconds: task_id=0, input=tail.0 > output=cloudwatch_logs.0 (out_id=0) [2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it. [2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status. [2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application not found. Will attempt to create it. [2024/05/17 03:47:54] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status. [2024/05/17 03:47:54] [error] [output:cloudwatch_logs:cloudwatch_logs.0] Failed to send events [2024/05/17 03:47:54] [ warn] [engine] failed to flush chunk '1-1715917669.177582045.flb', retry in 10 seconds: task_id=2, input=tail.1 > output=cloudwatch_logs.0 (out_id=0) [2024/05/17 03:47:55] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:55] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.eks-pod-identity-agent-q5vxk_kube-system_eks-pod-identity-agent-59bc7f4cc8194e140161525d8c2db02008ca5718770d120cfe7d48f6a1203a21.log [2024/05/17 03:47:55] [ info] [engine] flush chunk '1-1715917664.929877801.flb' succeeded at retry 1: task_id=0, input=tail.0 > output=cloudwatch_logs.0 (out_id=0) [2024/05/17 03:47:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application [2024/05/17 03:47:56] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-application.var.log.containers.fluent-bit-fbf8584f8-vvsnl_amazon-cloudwatch_fluent-bit-1deecf439298817933984983865ffed777bf4347c4b8d42a1562f215cc12468a.log [2024/05/17 03:47:56] [ info] [engine] flush chunk '1-1715917664.940218327.flb' succeeded at retry 1: task_id=1, input=tail.1 > output=cloudwatch_logs.0 (out_id=0) [2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Creating log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service in log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/dataplane [2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.systemd.kubelet.service [2024/05/17 03:47:59] [ info] [output:cloudwatch_logs:cloudwatch_logs.1] Created log stream ip-192-168-34-117.ca-central-1.compute.internal-dataplane.tail.var.log.containers.kube-proxy-pgx7n_kube-system_kube-proxy-5a8416e3845cdf77cbdc4789db4e1494b6959b912afb07c6d39158e1d41dbc61.log [2024/05/17 03:47:59] [ info] [engine] flush chunk '1-1715917664.948458095.flb' succeeded at retry 1: task_id=3, input=systemd.3 > output=cloudwatch_logs.1 (out_id=1) [2024/05/17 03:47:59] [ info] [engine] flush chunk '1-1715917664.974275925.flb' succeeded at retry 1: task_id=4, input=tail.4 > output=cloudwatch_logs.1 (out_id=1) [2024/05/17 03:48:04] [ info] [engine] flush chunk '1-1715917669.177582045.flb' succeeded at retry 1: task_id=2, input=tail.1 > output=cloudwatch_logs.0 (out_id=0)

PettitWesley avatar May 17 '24 04:05 PettitWesley

OperationAbortedException with log_group_class: Race Condition between Fluent Bit instances

Below is normal even if the log_group_class was successfully set.

[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Creating log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] CreateLogGroup API responded with error='OperationAbortedException', message='A conflicting operation is currently in progress against this resource. Please try again.'
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to create log group
[2024/05/17 03:47:49] [error] [output:cloudwatch_logs:cloudwatch_logs.1] Failed to send events

[2024/05/17 03:47:54] [ warn] [output:cloudwatch_logs:cloudwatch_logs.0] Log Group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application already exists; Fluent Bit did not create this group in this execution. Fluent Bit therefore was unable verify or set INFREQUENT_ACCESS storage. Check CloudWatch Console or API for the groups storage class status.

A k8s cluster usually has > 1 node and Fluent Bit is run as a daemonset. On startup, every daemon will try to set the logGroupClass, but only one can succeed and create the group. Thus, the above messages will be printed by individual Fluent Bit instances that failed.

The succeeding instance would print:

[2024/05/17 03:47:49] [ info] [output:cloudwatch_logs:cloudwatch_logs.0] Created log group infrequent-access-test16/containerinsights/Bottlerocket-Linux-v1-29/application with storage class INFREQUENT_ACCESS

Search CloudWatch Log Insights for the above to determine if one instance succeeded in setting the log group class because it created the group.

Workaround: Create Log Group before deployment

The best practice workaround is to create log groups before deployment; generally at cluster creation time. Each Fluent Bit instance will not need to create the group. You can disable group creation with auto_create_group Off in your Fluent Bit config.

PettitWesley avatar May 17 '24 04:05 PettitWesley 

[OUTPUT]
        Name                cloudwatch_logs
        Match               dataplane.*
        region              ${AWS_REGION}
        log_group_name      standard-access-test16/containerinsights/${CLUSTER_NAME}/dataplane
        log_stream_prefix   ${HOST_NAME}-
        auto_create_group   true
        extra_user_agent    container-insights
        log_group_class     STANDARD

PettitWesley avatar May 17 '24 16:05 PettitWesley

@PettitWesley thank you, can you pls add here the PR for docs ? (cc: @lecaros)

edsiper avatar May 29 '24 16:05 edsiper

ping @PettitWesley

@PettitWesley thank you, can you pls add here the PR for docs ? (cc: @lecaros)

lecaros avatar Jun 04 '24 22:06 lecaros

Hi,

When I updated 3.0.6 -> 3.0.7 I noticed my previously working config:

[OUTPUT]
        Name cloudwatch_logs
        Match app_errors/*
        region us-east-1
        log_group_name app_errors
        log_stream_prefix errors/
        log_format json
        Alias cloudwatch_logs_app_errors

Required me to add

        auto_create_group On

before fluent-bit would start. Could it be related to this PR?

michael-stevens avatar Jun 12 '24 15:06 michael-stevens

Echoing @michael-stevens , I believe this has introduced a regression. I do not want fluent bit to create my log groups on my behalf, so I do not want to enable auto_create_group.

nathanielks avatar Jun 17 '24 13:06 nathanielks