Feature Request: In-Memory Kafka Container for Cross-Language Testing

[alm0ra]

Summary:

I’d like to propose a feature enhancement for the mockafka library. The goal is to expand its functionality to act as a lightweight, in-memory Kafka container that can be used for testing purposes across different programming languages. This feature would enable developers to write and execute tests without needing a separate Kafka instance.

Background:

Currently, running Kafka for integration or end-to-end testing can be challenging, especially in a microservice environment where setting up and managing Kafka instances can be complex and resource-intensive. The mockafka library, while useful for faking Kafka interactions, primarily targets specific Kafka clients like aiokafka or confluent-kafka-python.

Proposed Enhancement:

In-Memory Kafka Container: Implement an in-memory Kafka instance that runs in a lightweight container. This container should:

Operate using Kafka's socket protocol, making it compatible with any Kafka client. Be capable of handling basic Kafka operations (e.g., producing and consuming messages) without requiring a full Kafka setup. Cross-Language Compatibility: Ensure that the container can be accessed from various programming languages, not just Python. This would involve:

Using a standard protocol that Kafka clients across different languages can interact with. Documenting the container’s usage and integration steps for various languages. Testing Integration: Provide mechanisms for:

Setting up and tearing down the container programmatically. Configuring the container to simulate different Kafka scenarios. Motivation:

The ability to use a lightweight, in-memory Kafka instance would greatly simplify testing in development environments, particularly for integration or end-to-end tests. It would eliminate the need for separate Kafka instances and streamline the testing setup process.

Challenges:

I am not familiar with Kafka’s socket programming or the protocol it uses, so I recognize that this may be a complex project. The implementation would need to handle Kafka’s protocol requirements and ensure compatibility across different clients.

Request for Assistance:

Given the complexity of this feature, any guidance or collaboration on the implementation would be greatly appreciated. If anyone has experience with Kafka's internal protocol or socket programming, your input would be invaluable.