feat(kafka): add apache/kafka and apache/kafka-native support

[strowk]

What does this PR do?

Adding support for apache/kafka and apache/kafka-native images into existing kafka module without breaking backward compatibility for users who might have used existing Confluent features.

Apache images are significantly faster, especially native one (~2-3 seconds vs 11-17 for example test).

Why is it important?

Put simply Kafka Native is blazingly fast. It is based on GraalVM in native mode, which compiles Java code into native binary. This translates into very fast starting time and starting time is what always important in tests (unlike in runtime) and what is normally very slow for Java. In my relatively simple tests on my relatively powerful machine, the improvement is from 16.6 seconds to 3.6 seconds. Translate this into say 100 tests and the improvement can be measured in minutes if not hours for big projects.

This change also can fix this issue due to changed way of starting the process: https://github.com/testcontainers/testcontainers-go/issues/2206 , so long as user would switch image to the one from Apache.

Related issues

• Relates #2206

How to test this PR

Explain here how this PR will be tested by the reviewer: commands, dependencies, steps, etc.

# pull images first
docker pull confluentinc/confluent-local:7.4.0
docker pull confluentinc/confluent-local:7.5.0
docker pull apache/kafka-native:4.0.1
docker pull apache/kafka:4.0.1
docker pull apache/kafka-native:3.9.1
docker pull apache/kafka:3.9.1

# run test for all those images
cd modules/kafka && go test -timeout 90s -count 1 -run '^TestKafka$' github.com/testcontainers/testcontainers-go/modules/kafka; cd -

Check out difference in start times:

cd modules/kafka && go test -timeout 30s -count 1 -run '^ExampleRun_confluentinc$' github.com/testcontainers/testcontainers-go/modules/kafka; cd -

cd modules/kafka && go test -timeout 30s -count 1 -run '^ExampleRun_apacheNative$' github.com/testcontainers/testcontainers-go/modules/kafka; cd -

cd modules/kafka && go test -timeout 30s -count 1 -run '^ExampleRun_apacheNotNative$' github.com/testcontainers/testcontainers-go/modules/kafka; cd -

Typical outcome:

These are benchmarks that only measure start and stop:

goos: linux
goarch: amd64
pkg: github.com/testcontainers/testcontainers-go/modules/kafka
cpu: AMD Ryzen 9 7940HS w/ Radeon 780M Graphics
=== RUN   BenchmarkConfluentStartStop
BenchmarkConfluentStartStop
BenchmarkConfluentStartStop-16                 1        13686264643 ns/op        9148552 B/op      50522 allocs/op
=== RUN   BenchmarkApacheNativeStartStop
BenchmarkApacheNativeStartStop
BenchmarkApacheNativeStartStop-16              2         882227018 ns/op         2650628 B/op      10643 allocs/op
=== RUN   BenchmarkApacheStartStop
BenchmarkApacheStartStop
BenchmarkApacheStartStop-16                    1        3708939089 ns/op         7082360 B/op      34003 allocs/op

Confluent: ~13.7 s Apache Native: ~0.9 s Apache: ~3.7 s

Additional Context

There are two other PR's attempting to add Kafka Native support.

Difference with 2683

• https://github.com/testcontainers/testcontainers-go/pull/2683

A brief difference:

• this PR keeps kafka module backwards-compatible to avoid potentially breaking workflows for users depending on details of confluent implementation
• this PR does not attempt to add SASL support and instead only focuses on using apache/kafka and apache/kafka-native images
• this PR does not remove starting script (advertising needs to detect correct hostname, at least for docker in Windows) and instead just adds "exec" before starting Apache Kafka process, this should help with related issue linked above because exec would replace process instead of starting a new one and piping streams, so signals would go to the right process properly

Difference with 3488

• https://github.com/testcontainers/testcontainers-go/pull/3488

A brief difference:

• this PR updates documentation in such a way that would steer new users towards most efficient apache/kafka-native image, but also is clear about which module version supports it and about tradeoffs (i.e missing CLI tools)
• this PR introduces three separate ExampleRun_.. methods in order to have three different "tabs" in documentation for every kind of supported image
• this PR does not change starterScriptContent or CMD for confluent case in any meaningful way, so has a chance of being more backwards compatible
• this PR keeps env variables such that docs still show them completely (or as completely as before)
• this PR does not add docker inspect call that would be used to build starter script
• this PR keeps two separate starter scripts - one for confluent and one for apache instead of making script smarter to pick what to do based on presence of confluent-related files (and documentation is changed accordingly to show both scripts depending on what user picks)
• this PR instead chooses starting script based on prefix of docker image - this might break, however, if user does not use "docker.io", but some different registry. In this case user has an option to override the choice: kafka.WithApacheFlavor() or kafka.WithConfluentFlavor() or even pass their own custom script using kafka.WithStarterScript

[netlify[bot]]

✅ Deploy Preview for testcontainers-go ready!

Name	Link
🔨 Latest commit	3034eea0e4763f58318c6736d790244cced5e488
🔍 Latest deploy log	https://app.netlify.com/projects/testcontainers-go/deploys/691efc1077bbca00089064f1
😎 Deploy Preview	https://deploy-preview-3249--testcontainers-go.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[strowk]

For comparison: on my machine currently, TestKafka in kafka_native runs in 3.6 seconds, while original module test runs in 16.6 seconds. On big codebases and weaker machines this could translate in significant speed up!

$ cd modules/kafka_native && go test -timeout 30s -count 1 -run '^TestKafka$' github.com/testcontainers/testcontainers-go/modules/kafka_native ; cd -
ok      github.com/testcontainers/testcontainers-go/modules/kafka_native        3.693s

$ cd modules/kafka && go test -timeout 30s -count 1 -run '^TestKafka$' github.com/testcontainers/testcontainers-go/modules/kafka ; cd -
ok      github.com/testcontainers/testcontainers-go/modules/kafka       16.654s

Tests themselves are identical, only different images and less confluent-dependant configs.

Upd: Commands above are outdated, check PR description for up to date way to measure.

[mdelapenya]

Hi @strowk thanks for adding this! I'm currently taking a look, also discussing internally with the other TC maintainers if this code should live as a new sub-package under the kafka module. What are your thoughts on this?

[strowk]

Sounds ok, though not sure what would interface look like

[mdelapenya]

Hmmm thinking more in depth about the submodule, will the existing kafka options apply to this native image? If so, then we should probably find them, although it can be probably done in a follow-up.

[mabrarov]

Hi @mdelapenya and @strowk,

Are we ensured we need a new module instead of extending existing one (kafka)? I planned to prepare that change - refer to https://github.com/testcontainers/testcontainers-go/compare/main...mabrarov:testcontainers-go:feat/kafka_alt_images - since May 2025 (refer to https://github.com/testcontainers/testcontainers-go/pull/2683#issuecomment-2912345737 and https://github.com/mabrarov/testcontainers-go-kafka-2206/pull/2), but I got time for that just recently.

Please note that (my case) usage of apache/kafka-native image can be problematic due to this image doesn't contain tools (like kafka-topics) which can be required for Kafka health check (e.g. in Docker Compose or Kubernetes). Using different images for tests (Testcontainers: apache/kafka-native) and local / dev or QA environments (Docker Compose, Kubernetes: apache/kafka) can bring additional issues.

Thank you.

[strowk]

@mabrarov , if I understand correctly some earlier concerns from maintainers were that removing confluent image from existing module would be not backwards compatible change for users that rely on something confluent specific (I for sure am one of those, albeit I know how to transition).

I.e it would for those users look such that the new version is pulled by something like dependabot or renovate and then they have to figure out how to transition to new image and have to stop updating testcontainers until the solution is found, which is somewhat undesirable situation. I base this line of thinking on the discussion in the other PR introducing kafka native that I linked in description.

https://github.com/testcontainers/testcontainers-go/pull/2683#issuecomment-2856920122

As to bitnami.. seeing what Broadcom did recently to images and how much the secured ones cost.. I sort of doubt it would find very many users TBH. This is just my opinion of course, but I would avoid bitnami as much as possible from now on.

[mabrarov]

Hi @strowk,

You are right about binami/kafka image (the reason I got back to this work).

As for confluentinc/confluent-local image - we can keep it too while adding support for apache/kafka and apache/kafka-native images to kafka module. Feel free to check feat/kafka_alt_images branch I referenced in my previous comment. Anyway, I'm happy (and appreciate) the work on kafka module is going on even if it will be a dedicated module. My need in that case is to support not just apache/kafka-native but apache/kafka image too.

Thank you.

[strowk]

@mabrarov

Ooh, thanks for clarification, I did not know that there separate apache/kafka is somehow significantly better than the native one.

This does mean that it would be better for users to have access to all three possible images -confluent/apache-not-native/apache-native via same module and pick between "flavors" by some sort of an option probably? I feel like it should be more explicit than just giving different image and module generating different configs based on that image.. Though of course configs derived from image would be easier to use probably, assuming it will not break :)

[mabrarov]

@strowk,

it would be better for users to have access to all three possible images -confluent/apache-not-native/apache-native via same module and pick between "flavors" by some sort of an option probably?

It is a question I have too. Maybe maintainers of Testcontainers for Go can answer it?

IMHO: only limited set of docker images (including tags) can be supported because we cannot predict what changes will be introduced in new version of image. Considering this limitation I'd just recommend to use specific (including tag) image with Testcontainers.

Thank you.

[strowk]

@mabrarov , I suppose that deriving what to do from parsing the image is a common answer in codebase as I can see:

https://github.com/testcontainers/testcontainers-go/blob/main/modules/elasticsearch/version.go

https://github.com/testcontainers/testcontainers-go/blob/ec51c6715cca41d30daacc7519118111a15ea623/modules/localstack/localstack.go#L32

https://github.com/testcontainers/testcontainers-go/blob/ec51c6715cca41d30daacc7519118111a15ea623/modules/redpanda/redpanda.go#L439

Though it is usually about version of same "flavor", but I dont see that much difference, except we probably would need to allow user to override whatever we define..

[mabrarov]

Hi @strowk,

Examples you referenced in https://github.com/testcontainers/testcontainers-go/pull/3249#issuecomment-3506835063 cover only lower bound while I am worried about upper bound too. Anyway, we can start from:

1. Check for the minimum version for each image repository/flavor.
2. Use some specific images in tests to have them as recommendation for the users (maybe need to cover that explicitly in documentation).

I feel it will be still the "Testcontainers way".

Thank you.

[strowk]

@mabrarov , I have made an attempt at this based on a very simple logic that looks like this:

const (
	apacheKafkaImagePrefix  = "apache/kafka"
	confluentincImagePrefix = "confluentinc/"
	dockerIoPrefix          = "docker.io/"
)

func isApache(image string) bool {
	return strings.HasPrefix(image, apacheKafkaImagePrefix) || strings.HasPrefix(image, dockerIoPrefix+apacheKafkaImagePrefix)
}

func isConfluentinc(image string) bool {
	return strings.HasPrefix(image, confluentincImagePrefix) || strings.HasPrefix(image, dockerIoPrefix+confluentincImagePrefix)
}

func getStarterScriptContent(image string) string {
	if isApache(image) {
		return apacheStarterScriptContent
	} else if isConfluentinc(image) {
		return confluentincStarterScriptContent
	} else {
		// Default to confluentinc for backward compatibility
		// in situations when image was custom specified based on confluentinc
		return confluentincStarterScriptContent
	}
}

This does not care about specific version as such, only whether or not image is confluent vs apache. In this "apache/kafka-native" and "apache/kafka" would work the same (starter script has no issues with either so far as I can tell).

It seems to work on and I tried with confluent, native and usual apache with this times:

• 14.77s for confluentinc
• 2.13s for native
• 5.65s for apache not native

This is just one run of course, so not super representative, but kind of interesting that apache/kafka shows good timing vs confluentinc one, though still not quite as good as native.

[coderabbitai[bot]]

[!WARNING]

Rate limit exceeded

@strowk has exceeded the limit for the number of commits or files that can be reviewed per hour. Please wait 17 minutes and 31 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

📥 Commits

Reviewing files that changed from the base of the PR and between 1fc9b85fa1edbfa10461e3cb3b5381302e863df0 and 3034eea0e4763f58318c6736d790244cced5e488.

📒 Files selected for processing (2)

• docs/modules/kafka.md (5 hunks)
• modules/kafka/options_test.go (1 hunks)

Summary by CodeRabbit

• New Features

  • Added support for Apache Kafka and Apache Kafka Native image variants alongside Confluent images.
  • Introduced flavor selection options for Apache and Confluent Kafka configurations.
  • Added custom starter script configuration capability.
  • Added localhost listener configuration support.

• Documentation

  • Updated Kafka module documentation with comprehensive guidance for multiple image types, flavor options, and KRaft mode requirements.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

Detect Kafka image family at runtime and introduce RunOption-based configuration (WithApacheFlavor, WithConfluentFlavor, WithStarterScript, WithClusterID); select appropriate starter script during post-start; update Run flow, docs, examples, tests, and benchmarks to support multiple image flavors, localhost listener, and graceful shutdown.

Changes

Cohort / File(s)	Summary
Documentation docs/modules/kafka.md	Title/intro and examples updated; added Apache image support (apache/kafka, apache/kafka-native), starter-script notes, Kraft requirements, localhost listener, and Run signature showing testcontainers.ContainerCustomizer.
Core runtime & starter script modules/kafka/kafka.go	Starter script refactor: separate Apache/Confluent starter scripts, use path-based copy; Run builds runOptions from image+opts; copyStarterScript now accepts *runOptions; validation uses image-family detection.
Options surface modules/kafka/options.go	New RunOption type implementing ContainerCustomizer; added WithStarterScript, WithClusterID, WithApacheFlavor, WithConfluentFlavor; flavor conflict detection and starter-script precedence implemented.
Image detection & selection modules/kafka/version.go	New image prefix constants and predicates (isApache, isConfluentinc) with docker.io handling; getStarterScriptContent(image) selects starter script based on image or opts.
Examples modules/kafka/examples_test.go	Added examples for apache native/non-native, flavor-specific usage, and localhost-listener; previous example renamed to ExampleRun_confluentinc.
Unit tests (options/version) modules/kafka/options_test.go, modules/kafka/version_test.go	Tests for starter-script selection, isApache/isConfluentinc predicates, and WithStarterScript override behavior.
Integration & behavior tests modules/kafka/kafka_test.go	Converted to table-driven tests across image variants; added TestKafkaGracefulShutdown, TestKafkaLocalhostListener, TestFailOnBothFlavors, and helper runner.
Benchmarks modules/kafka/benchmark_test.go	Added start/stop benchmarks for Confluent and Apache (native and non-native) images.
API/signature changes modules/kafka/kafka.go, modules/kafka/options.go	copyStarterScript signature changed to accept (*runOptions, ...); Run/docs updated to accept ...testcontainers.ContainerCustomizer; new RunOption helpers added; WithClusterID moved/adjusted.

Sequence Diagram(s)

sequenceDiagram
    participant Caller as Caller (examples/tests)
    participant Run as kafka.Run
    participant Resolver as runOptions
    participant Post as post-start hook
    participant Copy as copyStarterScript
    participant Container as Container FS

    Note over Run,Resolver: build runOptions from image + supplied RunOption(s)
    Caller->>Run: Run(ctx, image, opts...)
    Run->>Resolver: build runOptions(image, opts)
    Run->>Post: register post-start(container, runOptions)
    Post->>Copy: copyStarterScript(ctx, runOptions, container)
    Copy->>Resolver: resolve starter script (getStarterScriptContent / opts)
    alt WithStarterScript supplied
        Resolver-->>Copy: custom starter script
    else Apache image detected
        Resolver-->>Copy: apache starter script
    else
        Resolver-->>Copy: confluent starter script (default)
    end
    Copy->>Container: write starter script path & set entrypoint
    Container-->>Copy: ack
    Post-->>Run: readiness & wait completion
    Run-->>Caller: container ready / error

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

• Files needing extra attention:
  • modules/kafka/version.go — prefix matching and docker.io variants.
  • Propagation of RunOption/runOptions and updated copyStarterScript(ctx, opts, c) signature across call sites.
  • Precedence and conflict handling between WithStarterScript, WithApacheFlavor, and WithConfluentFlavor.
  • Tests relying on timing/waits (graceful shutdown, localhost listener) for flakiness and cleanup.
  • validateKRaftVersion change to image-family detection.

Possibly related PRs

• testcontainers/testcontainers-go#3414 — overlaps on Kafka Run flow, starter-script/post-start handling, and option pipeline changes.
• testcontainers/testcontainers-go#3325 — similar migration to ContainerCustomizer/Run-style API and module option surfaces.
• testcontainers/testcontainers-go#3276 — touches Kafka KRaft/version validation logic related to image checks.

Suggested reviewers

• mdelapenya

Poem

🐇 I hopped through files with whiskers bright,
I pick a script by image, day or night,
Apache or Confluent — starter chosen right,
Brokers spin, listeners hum into light,
I leave a tiny carrot for CI's delight.

Pre-merge checks and finishing touches

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 26.67% which is insufficient. The required threshold is 80.00%.	You can run @coderabbitai generate docstrings to improve docstring coverage.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The pull request title clearly and concisely describes the main change: adding support for apache/kafka and apache/kafka-native images to the existing kafka module.
Description check	✅ Passed	The pull request description thoroughly explains what is being added (Apache Kafka image support), why it matters (significant performance improvements), how to test it (with specific commands and expected results), and related considerations.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[mabrarov]

Hi @strowk,

Just to complete my work with some milestone I opened #3488. Please feel free to discuss with maintainers what way to proceed - copy some parts of that PR (e.g. links to GitHub issues, test for graceful shutdown of Kafka) into this PR (you was the first 😄) or just complete with this PR... or maybe complete/merge #3488 instead of this PR.

Thank you for your help and work.

[strowk]

Hey @mdelapenya , I have made a lot of changes to this:

• new stuff is all added into existing "kafka" module, "kafka_native" is removed
• user just needs to give apache/kafka-native or apache/kafka image (those are all explained in docs)
• I had to merge from main, due to conflicts, tell me if you'd rather I made another PR (contributing doc says that force push is a no-no).

@mabrarov , as requested this should work for "apache/kafka" as well as for "apache/kafka-native" , there are tests and examples to confirm.