Critical Security Vulnerability: Fluent Bit (CVE-2024-4323)

[gsmith-sas]

trafficstars

Introduction

A significant security vulnerability has been identified in Fluent Bit, one of the open-source components deployed as part of this project. Within this project, Fluent Bit is used to collect both the Kubernetes events and the log messages generated by the pods running on the cluster. Fluent Bit has published a blog post with more information on the vulnerability.

Fluent Bit has released an update to address this vulnerability and we are updating our project to leverage this new version of Fluent Bit. Our initial testing results, so far, have been very positive with no problems identified with incorporating this new version of Fluent Bit. However, we need to complete additional testing before releasing a new version of SAS Viya Monitoring that includes this new version of Fluent Bit.

In the meantime, organizations especially concerned about this vulnerability may choose to upgrade the version of Fluent Bit deployed in their environments on their own. Please remember that we are still completing our testing and it is possible that some incompatibility with this version of Fluent Bit will be discovered. However, if you are comfortable accepting that risk, you can manually upgrade using the steps outlined below.

If an incompatibility is identified, we will update this issue to share that information. When a new version of SAS Viya Monitoring is released that incorporates the patched version of Fluent Bit, we will share this information here as well.

Preparation

• The following steps assume you have no existing customizations or configuration tweaks related to SAS Viya Monitoring in place, i.e. you have no existing USER_DIR directory. If you do have an existing USER_DIR directory, you should make these changes in the corresponding files in your USER_DIR directory (if they exist), or create the corresponding files there (if they do not already exist).
• Create a directory to hold the customization files. This directory should exist outside of your cloned copy of the project repo. In the following example, we are creating this directory under our home directory: mkdir ~/user-dir-fbsecurityfix
• Create an environment variable pointing to that directory export USER_DIR=~/user-dir-fbsecurityfix
• Create the logging subdirectory mkdir $USER_DIR/logging
• Create the file $USER_DIR/logging/user-values-fluent-bit-events.yaml. This file contains customizations related to the Fluent Bit instance used to collect Kubernetes Events. This file should contain the following lines. Note that this is a YAML file and, therefore, is very sensitive to indentation. The 2nd line in this file should be indented two spaces.

image:
  tag: 3.0.4

• Create the file $USER_DIR/logging/user-values-fluent-bit-opensearch.yaml. This file contains customizations related to the Fluent Bit instance used to collect log messages from the pods/containers running on the cluster. This file should contain same 2 lines shown above. Therefore, it may be easier to just copy that file. cp $USER_DIR/logging/user-values-fluent-bit-events.yaml $USER_DIR/logging/user-values-fluent-bit-opensearch.yaml

Updating an EXISTING deployment

• If you are updating an existing deployment of SAS Viya Monitoring, you only need to redeploy the two Fluent Bit instances.
• Redeploy the Fluent Bit instance used to collect Kubernetes Events. ./logging/bin/deploy_fluentbit_k8sevents_opensearch.sh
• Redeploy the Fluent Bit instance used to collect log messages. ./logging/bin/deploy_fluentbit_opensearch.sh
• Once the pods have all be redeployed and restarted, you can confirm that you are running the patched version of Fluent Bit by checking the container images of the running pods. For example, the following output shows that Fluent Bit version 3.0.4 is running on all 5 Fluent Bit pods collecting log messages running in a test cluster:

$ kubectl -n logging describe pod -l app.kubernetes.io/instance=v4m-fb |grep "Image:"
    Image:         cr.fluentbit.io/fluent/fluent-bit:3.0.4
    Image:         cr.fluentbit.io/fluent/fluent-bit:3.0.4
    Image:         cr.fluentbit.io/fluent/fluent-bit:3.0.4
    Image:         cr.fluentbit.io/fluent/fluent-bit:3.0.4
    Image:         cr.fluentbit.io/fluent/fluent-bit:3.0.4

Deploying a NEW deployment

• If you are deploying a new instance of SAS Viya Monitoring, you can now deploy all of the log components (e.g. Fluent Bit, OpenSearch, OpenSearch Dashboards, etc.) using the primary deployment script. ./logging/bin/deploy_logging.sh