fluent-bit icon indicating copy to clipboard operation
fluent-bit copied to clipboard
verified publisher icon fluent

out_azure_blob: add zstd compression support

Open nberlee opened this issue 3 weeks ago • 3 comments

Enter [N/A] in the box, if an item is not applicable to your change.

Testing Before we can approve your change; please submit the following in a comment:

  • [x] Example configuration file for the change
[SERVICE]
    flush     1
    log_level info

[INPUT]
    name      dummy
    dummy     {"name": "Fluent Bit", "year": 2020}
    samples   1
    tag       app.log


[OUTPUT]
    name                  azure_blob
    match                 *
    account_name          devstoreaccount1
    shared_key            Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==
    path                  devstoreaccount1
    container_name        devstoreaccount1
    auto_create_container on
    tls                   off
    emulator_mode         on
    compress              zstd
    compress_blob         true
    blob_type             blockblob
    endpoint              http://localhost:10000
  • [x] Debug log output from testing the change debug.log

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • [N/A] Run local packaging test showing all targets (including any new ones) build.
  • [N/A ] Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • [x] Documentation required for this feature

Backporting

  • [ ] Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • New Features

    • Added ZSTD compression option for Azure Blob uploads alongside GZIP; configurable per upload.

  • Improvements

    • Unified compression handling with clearer fallbacks and descriptive logging; content-type/encoding reflect applied compression.

  • Documentation

    • Updated compression option help text to describe supported algorithms and behavior.

  • Tests

    • Added unit tests covering compression behavior and HTTP header/content assertions.

✏️ Tip: You can customize this high-level summary in your review settings.

nberlee avatar Nov 25 '25 08:11 nberlee

[!NOTE]

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

Walkthrough

Adds configurable payload compression (gzip and zstd) to the Azure Blob output plugin: replaces a gzip-only flag with a compression enum, integrates compression into payload/URI/header flows, updates HTTP APIs to carry content-encoding, and adds runtime tests for compression behavior.

Changes

Cohort / File(s) Summary
Core compression & payload flow
plugins/out_azure_blob/azure_blob.c		 Adds azure_blob_compress_payload() and azure_blob_compression_name(); replaces compress_gzip with compression enum; manages network vs blob compression state, attempts compression and falls back to uncompressed on failure, sets content-type/encoding flags, and expands logging.
Configuration & init
plugins/out_azure_blob/azure_blob_conf.c		 Replaces compress_gzip with compression (FLB_COMPRESSION_ALGORITHM), parses/validates NONE/GZIP/ZSTD, errors on invalid values, and includes compression header.
Public headers / macros
plugins/out_azure_blob/azure_blob.h		 Adds AZURE_BLOB_CT_ZSTD and AZURE_BLOB_CE_ZSTD macros; renames struct member compress_gzip → compression; declares azb_select_compression_strategy().
URI / block-blob helpers
plugins/out_azure_blob/azure_blob_blockblob.c		 Adds azb_blob_extension() to return extension based on compression ("", ".gz", ".zst") and refactors URI generation to use it.
HTTP integration
plugins/out_azure_blob/azure_blob_http.c, plugins/out_azure_blob/azure_blob_http.h		 Maps ZSTD to Content-Type/Content-Encoding, emits application/zstd/zstd headers, and updates azb_http_client_setup / azb_http_canonical_request signatures to include content_encoding.
Tests & build entries
tests/runtime/CMakeLists.txt, tests/runtime/out_azure_blob_compression.c		 Adds runtime test source and CMake entry; tests block-blob extension selection and HTTP header behavior for zstd, gzip, and disabled compression.

Sequence Diagram(s)

sequenceDiagram
    participant Plugin as Azure Blob Plugin
    participant Compressor as flb_compression (gzip/zstd)
    participant HTTP as HTTP Client
    participant Azure as Azure Blob Storage

    Note over Plugin,Compressor: Decide compression strategy (config + blob type)
    Plugin->>Compressor: compress(payload, algorithm)
    alt compression succeeds
        Compressor-->>Plugin: compressed_payload
        Plugin->>HTTP: create request(payload=compressed_payload, Content-Encoding, Content-Type, URI ext)
    else compression fails or disabled
        Compressor-->>Plugin: error / no-op
        Plugin->>HTTP: create request(payload=uncompressed, default headers, URI ext="")
    end
    HTTP->>Azure: send PUT/POST (headers + payload)
    Azure-->>HTTP: response
    HTTP-->>Plugin: response
    Note over Plugin,Azure: Plugin logs outcome and updates compression-applied flags

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

  • Verify parsing/validation of compression in azure_blob_conf.c.
  • Inspect compression application, fallback logic, and state flags in azure_blob.c.
  • Confirm updated header mapping and modified function signatures in azure_blob_http.{c,h} and all callsites.
  • Review new tests in tests/runtime/out_azure_blob_compression.c for expected coverage.

Possibly related PRs

  • fluent/fluent-bit#10697 — Introduces FLB_COMPRESSION_ALGORITHM_ZSTD and related compression infrastructure used by this plugin.
  • fluent/fluent-bit#10833 — Adjusts flb_zstd APIs/return shapes; directly affects zstd callsites in this plugin.

Suggested reviewers

  • edsiper
  • koleini
  • fujimotos
  • PettitWesley

Poem

🐰 I nudged a payload, chose a zip,

gzip sang low, zstd danced quick—flip!
Extension set, headers in tow,
Off to Azure the small bytes go.
Hop! The blobs are snug and hip.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)
Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 9.09% which is insufficient. The required threshold is 80.00%. You can run @coderabbitai generate docstrings to improve docstring coverage.
✅ Passed checks (2 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title accurately summarizes the primary change: adding ZSTD compression support to the azure_blob output plugin, which aligns with the substantial refactoring of compression handling throughout the codebase.
✨ Finishing touches
  • [ ] 📝 Generate docstrings
🧪 Generate unit tests (beta)
  • [ ] Create PR with unit tests
  • [ ] Post copyable unit tests in a comment
📜 Recent review details

Configuration used: defaults

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 85d54b94245ce27ab130303132c6e6859d697369 and b7fd6a3ef2728badd0931698909b2f289f50d721.

📒 Files selected for processing (8)
  • plugins/out_azure_blob/azure_blob.c (7 hunks)
  • plugins/out_azure_blob/azure_blob.h (3 hunks)
  • plugins/out_azure_blob/azure_blob_blockblob.c (5 hunks)
  • plugins/out_azure_blob/azure_blob_conf.c (2 hunks)
  • plugins/out_azure_blob/azure_blob_http.c (3 hunks)
  • plugins/out_azure_blob/azure_blob_http.h (1 hunks)
  • tests/runtime/CMakeLists.txt (1 hunks)
  • tests/runtime/out_azure_blob_compression.c (1 hunks)
🚧 Files skipped from review as they are similar to previous changes (1)
  • plugins/out_azure_blob/azure_blob_http.c
🧰 Additional context used
🧠 Learnings (12)
📓 Common learnings 
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.
📚 Learning: 2025-08-29T06:25:27.250Z 
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
  • plugins/out_azure_blob/azure_blob_conf.c
  • plugins/out_azure_blob/azure_blob.h
  • plugins/out_azure_blob/azure_blob_blockblob.c
  • tests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:24:26.170Z 
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
  • plugins/out_azure_blob/azure_blob_conf.c
  • plugins/out_azure_blob/azure_blob.h
  • plugins/out_azure_blob/azure_blob_blockblob.c
  • tests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:25:27.250Z 
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
  • plugins/out_azure_blob/azure_blob_conf.c
  • plugins/out_azure_blob/azure_blob.h
  • plugins/out_azure_blob/azure_blob_blockblob.c
  • tests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:24:55.855Z 
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
  • plugins/out_azure_blob/azure_blob_conf.c
  • plugins/out_azure_blob/azure_blob.h
  • plugins/out_azure_blob/azure_blob_blockblob.c
  • tests/runtime/out_azure_blob_compression.c
📚 Learning: 2025-08-29T06:25:02.561Z 
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
  • plugins/out_azure_blob/azure_blob_conf.c
  • plugins/out_azure_blob/azure_blob.h
  • plugins/out_azure_blob/azure_blob_blockblob.c
📚 Learning: 2025-08-29T06:24:44.797Z 
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
  • plugins/out_azure_blob/azure_blob_conf.c
  • plugins/out_azure_blob/azure_blob_blockblob.c
📚 Learning: 2025-08-31T12:46:11.940Z 
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit CMakeLists.txt, the system library preference flags are defined as FLB_PREFER_SYSTEM_LIB_ZSTD and FLB_PREFER_SYSTEM_LIB_KAFKA with the FLB_ prefix.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
  • plugins/out_azure_blob/azure_blob_blockblob.c
📚 Learning: 2025-11-21T06:23:29.770Z 
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11171
File: include/fluent-bit/flb_lib.h:52-53
Timestamp: 2025-11-21T06:23:29.770Z
Learning: In Fluent Bit core (fluent/fluent-bit repository), function descriptions/documentation are not required for newly added functions in header files.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
  • plugins/out_azure_blob/azure_blob_blockblob.c
📚 Learning: 2025-08-31T12:46:11.940Z 
Learnt from: ThomasDevoogdt
Repo: fluent/fluent-bit PR: 9277
File: .github/workflows/pr-compile-check.yaml:147-151
Timestamp: 2025-08-31T12:46:11.940Z
Learning: In fluent-bit, the correct CMake flag for using system librdkafka is `FLB_PREFER_SYSTEM_LIB_KAFKA=ON`.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-09-08T11:21:33.975Z 
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 10851
File: include/fluent-bit/flb_simd.h:60-66
Timestamp: 2025-09-08T11:21:33.975Z
Learning: Fluent Bit currently only supports MSVC compiler on Windows, so additional compiler compatibility guards may be unnecessary for Windows-specific code paths.

Applied to files:

  • plugins/out_azure_blob/azure_blob.c
📚 Learning: 2025-12-08T05:21:45.014Z 
Learnt from: cosmo0920
Repo: fluent/fluent-bit PR: 11250
File: src/ripser/flb_ripser_wrapper.cpp:73-106
Timestamp: 2025-12-08T05:21:45.014Z
Learning: In the TDA processor (processor_tda) for Fluent Bit, the dimension limit FLB_RIPSER_MAX_BETTI_DIM is intentionally capped at 3 because the plugin uses embed_dim=3 and delay=1 in practice, and computing higher dimensions (>4) would be computationally prohibitive. The 8-slot betti array allocation is conservative headroom.

Applied to files:

  • plugins/out_azure_blob/azure_blob_blockblob.c
🧬 Code graph analysis (4)
plugins/out_azure_blob/azure_blob.c (2)
src/flb_gzip.c (1)
  • flb_gzip_compress (157-252)
src/flb_zstd.c (1)
  • flb_zstd_compress (33-57)
plugins/out_azure_blob/azure_blob_conf.c (1)
src/flb_output.c (1)
  • flb_output_get_property (1108-1111)
plugins/out_azure_blob/azure_blob.h (1)
plugins/out_azure_blob/azure_blob.c (1)
  • azb_select_compression_strategy (360-389)
tests/runtime/out_azure_blob_compression.c (3)
plugins/out_azure_blob/azure_blob_blockblob.c (1)
  • azb_block_blob_uri (72-121)
plugins/out_azure_blob/azure_blob_http.c (1)
  • azb_http_client_setup (299-382)
plugins/out_azure_blob/azure_blob.c (1)
  • azb_select_compression_strategy (360-389)
🔇 Additional comments (23)
plugins/out_azure_blob/azure_blob_blockblob.c (2)

35-46: LGTM! Clean helper for extension selection.

The static helper correctly returns string literals based on compression settings, and the const char * return type is appropriate. The fallback to .gz when compress_blob is enabled but no specific algorithm is set maintains backward compatibility.

77-92: Correct use of const char * for the extension.

The variable type matches the return type of azb_blob_extension(), which returns string literals. This aligns with the const-correctness fix noted in the PR comments.

tests/runtime/CMakeLists.txt (1)

207-207: LGTM! Test registration follows established patterns.

The new runtime test is correctly added using the FLB_RT_TEST macro with the FLB_OUT_AZURE_BLOB build flag, consistent with other output plugin tests in this section.

plugins/out_azure_blob/azure_blob.h (3)

33-39: LGTM! Clean extension of content type and encoding constants.

The new AZURE_BLOB_CT_ZSTD and AZURE_BLOB_CE_ZSTD constants follow the existing numbering scheme and maintain consistency with the existing gzip constants.

59-60: LGTM! Field rename supports multi-algorithm compression.

Renaming compress_gzip to compression (as an int representing the compression algorithm enum) correctly generalizes the field to support both gzip and zstd. The compress_blob field remains to control whether blob-level compression is applied.

171-174: LGTM! Function declaration matches implementation.

The declaration aligns with the implementation in azure_blob.c (lines 359-388), which correctly determines compression algorithm and blob compression preference based on context settings.

tests/runtime/out_azure_blob_compression.c (8)

42-74: LGTM! Proper fixture initialization with cleanup on failure.

The fixture initialization correctly handles error paths, cleaning up previously allocated resources before returning. The upstream and connection setup follows the pattern needed for HTTP client testing.

123-142: LGTM! ZSTD extension test correctly validates URI construction.

The test properly sets up the context with compress_blob = FLB_TRUE and compression = FLB_COMPRESSION_ALGORITHM_ZSTD, then verifies that the generated URI contains the .zst extension.

144-166: LGTM! Gzip fallback test validates backward compatibility.

The comment correctly documents that gzip remains the fallback when compress_blob is enabled but no explicit algorithm is configured, preserving legacy behavior.

168-188: LGTM! Disabled compression test validates no extension is added.

The test correctly verifies that when compress_blob = FLB_FALSE, no compression extension (.gz or .zst) is added to the URI, regardless of the compression setting.

190-244: LGTM! HTTP headers test validates Content-Encoding for ZSTD.

The test correctly uses AZURE_BLOB_AUTH_SAS to bypass signature generation (which would require decoded_sk), then verifies that Content-Encoding: zstd is set while Content-Type remains application/json.

246-293: LGTM! Content-Type test validates ZSTD MIME type.

The test correctly verifies that Content-Type: application/zstd is set when AZURE_BLOB_CT_ZSTD is passed, with no content encoding applied (AZURE_BLOB_CE_NONE).

295-336: LGTM! Comprehensive test of compression strategy logic.

The test covers all four combinations of compress_blob and compression settings, verifying:

  1. Explicit algorithm with blob compression → uses specified algorithm
  2. No algorithm with blob compression → falls back to gzip
  3. Explicit algorithm without blob compression → uses specified algorithm
  4. No algorithm without blob compression → no compression

338-346: LGTM! TEST_LIST correctly registers all test cases.

All six test functions are properly registered in the test list with descriptive names, and the list is correctly null-terminated.

plugins/out_azure_blob/azure_blob_http.h (1)

27-35: API extension for content encoding support correctly implemented.

All function signatures and call sites have been updated. The content_encoding parameter is consistently passed across all 10 azb_http_client_setup calls and the implementation call to azb_http_canonical_request, with appropriate encoding constants or variables used throughout the codebase.

plugins/out_azure_blob/azure_blob_conf.c (2)

23-23: LGTM!

The inclusion of flb_compression.h provides the FLB_COMPRESSION_ALGORITHM_* enum constants needed for the new compression configuration logic.

659-673: LGTM!

The compression configuration parsing correctly:

  • Defaults to FLB_COMPRESSION_ALGORITHM_NONE when not specified
  • Supports both gzip and zstd with case-insensitive matching
  • Provides a clear error message listing supported values on invalid input
  • Aborts initialization on invalid configuration, preventing runtime issues

The updated comment "Compress payload over the wire" accurately reflects the broader compression strategy that now supports multiple algorithms.

plugins/out_azure_blob/azure_blob.c (6)

27-28: LGTM!

The compression and ZSTD headers are correctly included to support the new multi-algorithm compression functionality. Based on learnings, ZSTD is a core bundled dependency in Fluent Bit and does not require conditional compilation guards.

141-143: LGTM!

The conditional null terminator addition is a minor optimization: when compression is configured, the null terminator is omitted since the buffer will be compressed to binary data later in http_send_blob, avoiding the overhead of including the terminator in the compression payload.

154-184: LGTM!

The compression helper functions provide clean abstractions:

  • azure_blob_compress_payload: Encapsulates algorithm-specific compression calls with consistent error semantics (0 on success, negative on failure), enabling graceful fallback to uncompressed payloads when compression fails.

  • azure_blob_compression_name: Provides human-readable algorithm labels for logging and error messages.

Both functions follow Fluent Bit conventions and handle edge cases appropriately.

360-389: LGTM!

The azb_select_compression_strategy function correctly implements the documented compression precedence:

  • When compress_blob is enabled, blob compression is preferred and uses the configured compress algorithm (defaulting to gzip if unspecified)
  • Otherwise, network-level compression uses the configured compress algorithm
  • The prefer_blob_compression flag clearly distinguishes between blob compression (compressed content stored) and network compression (transport encoding only)

The logic aligns with the configuration documentation and handles all configuration combinations correctly.

431-485: LGTM!

The compression logic correctly implements the dual-mode compression strategy:

Compression execution (lines 431-461):

  • Determines strategy via azb_select_compression_strategy
  • Attempts compression and gracefully falls back to uncompressed payload on failure
  • Sets exactly one flag (blob_compression_applied OR network_compression_applied) based on which mode was actually applied, addressing the prior review concern about both flags being set incorrectly

HTTP header mapping (lines 467-485):

  • Blob compression sets Content-Type (e.g., application/zstd) to indicate compressed blob content
  • Network compression sets Content-Encoding (e.g., zstd) with Content-Type: application/json to indicate transport-level compression
  • Correct precedence: blob compression takes priority since both cannot be advertised simultaneously

The implementation properly distinguishes between storing compressed blobs versus using transport compression, ensuring Azure correctly interprets the uploaded content.

1886-1896: LGTM!

The configuration property descriptions accurately document the compression functionality:

  • compress: Clearly specifies network transfer compression with both supported algorithms (gzip, zstd)
  • compress_blob: Correctly describes the default fallback (gzip), codec inheritance from compress when set, and the appendblob incompatibility

The descriptions match the implementation in azb_select_compression_strategy and the validation logic in the configuration module.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

coderabbitai[bot] avatar Nov 25 '25 08:11 coderabbitai[bot]

@coderabbitai Fixes for both nitpicks landed in the latest commit a2855d23c9d3:

  • azure_blob_blockblob.c: ext is now declared as const char *, matching the azb_blob_extension() return type.
  • azure_blob.c: the gzip-default test includes a brief comment noting the fallback behavior when no explicit compression algorithm is set.

nberlee avatar Nov 26 '25 13:11 nberlee

[!TIP] For best results, initiate chat on the files or code changes.

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Fetch the specific commit to verify the changes
git show a2855d23c9d3 --stat
echo "---"
# Show the changes in azure_blob_blockblob.c
git show a2855d23c9d3 -- plugins/out_azure_blob/azure_blob_blockblob.c | head -60
echo "---"
# Show the changes in azure_blob.c (looking for test comment)
git show a2855d23c9d3 -- plugins/out_azure_blob/azure_blob.c | grep -A5 -B5 "gzip-default\|fallback"

Length of output: 2392

🏁 Script executed:

#!/bin/bash
# Show the full diff for azure_blob.c to find the test comment
git show a2855d23c9d3 -- plugins/out_azure_blob/azure_blob.c | grep -A10 -B10 "test\|Test\|TEST" | head -100
echo "---"
# Also check tests/internal/azure_blob.c for the gzip default test comment
git show a2855d23c9d3 -- tests/internal/azure_blob.c | grep -A10 -B2 "gzip.*default\|fallback\|compression.*none"

Length of output: 1040

@nberlee Perfect! I've verified both fixes in commit a2855d23c9d3:

  1. ✅ azure_blob_blockblob.c: The ext variable is now properly declared as const char *, matching the return type of azb_blob_extension().

  2. ✅ tests/internal/azure_blob.c: The test_block_blob_extension_gzip_default test now includes a clear comment explaining the fallback behavior: "When no explicit algorithm is configured, gzip remains the fallback to preserve legacy behavior."

Both nitpicks have been properly addressed. Thank you for the quick fixes!

🧠 Learnings used 
Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components such as ARROW/PARQUET (which use `#ifdef FLB_HAVE_ARROW` guards), ZSTD support is always available and doesn't need build-time conditionals. ZSTD headers are included directly without guards across multiple plugins and core components.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:93-107
Timestamp: 2025-08-29T06:25:27.250Z
Learning: In Fluent Bit, ZSTD compression is enabled by default and is treated as a core dependency, not requiring conditional compilation guards like `#ifdef FLB_HAVE_ZSTD`. Unlike some other optional components, ZSTD support is always available and doesn't need build-time conditionals.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:39-42
Timestamp: 2025-08-29T06:24:26.170Z
Learning: In Fluent Bit, ZSTD compression support is enabled by default and does not require conditional compilation guards (like #ifdef FLB_HAVE_ZSTD) around ZSTD-related code declarations and implementations.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:52-56
Timestamp: 2025-08-29T06:24:55.855Z
Learning: ZSTD compression is always available in Fluent Bit and does not require conditional compilation guards. Unlike Arrow/Parquet which use #ifdef FLB_HAVE_ARROW guards, ZSTD is built unconditionally with flb_zstd.c included directly in src/CMakeLists.txt and a bundled ZSTD library at lib/zstd-1.5.7/.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: tests/internal/aws_compress.c:7-7
Timestamp: 2025-08-29T06:25:02.561Z
Learning: In Fluent Bit, ZSTD (zstandard) compression library is bundled directly in the source tree at `lib/zstd-1.5.7` and is built unconditionally as a static library. Unlike optional external dependencies, ZSTD does not use conditional compilation guards like `FLB_HAVE_ZSTD` and is always available. Headers like `<fluent-bit/flb_zstd.h>` can be included directly without guards.

Learnt from: shadowshot-x
Repo: fluent/fluent-bit PR: 10794
File: src/aws/flb_aws_compress.c:26-26
Timestamp: 2025-08-29T06:24:44.797Z
Learning: In Fluent Bit, ZSTD support is always available and enabled by default. The build system automatically detects and uses either the system libzstd library or builds the bundled ZSTD version. Unlike other optional dependencies like Arrow which use conditional compilation guards (e.g., FLB_HAVE_ARROW), ZSTD does not require conditional includes or build flags.

coderabbitai[bot] avatar Nov 26 '25 13:11 coderabbitai[bot]

@codex review

nberlee avatar Dec 17 '25 15:12 nberlee

@codex review

nberlee avatar Dec 17 '25 16:12 nberlee

Codex Review: Didn't find any major issues. Breezy!

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector[bot] avatar Dec 17 '25 16:12 chatgpt-codex-connector[bot]