awesome-kafka

Everything about Apache Kafka

Share your Kafka ❤️ through pull requests :)

What can you expect to see here?

• Development
  • Client libraries
  • Kafka Streams libraries
  • KSQL
  • Connectors
  • Producers
  • Consumers
  • Transformations
  • Testing
• Operations
  • CI/CD
  • Operational Utilities
  • Monitoring
  • Performance tools
  • Security
  • Audit
  • Mirroring
  • Backup
  • Tools
  • Metadata Management
  • Schema Management
  • UI
• Deployment
  • Kubernetes
  • Docker
  • Integrations
  • Alternative Implementations
• Project built on top of Kafka
• Unsorted

Development

Client libraries

• Go
• Go
• Go
• Go Kafka streaming
• Node
• Node
• Node
• Node Kafka streams
• Node librdkafka
• Php
• Php
• Perl
• Python (asyncio)
• Python (kafka-python)
• Python (pykafka)
• Ruby
• Phobos - Ruby App framework
• Karafka - Ruby and Rails framework
• Dart
• JRuby
• Elixir
• Erlang
• Rust
• LinkedIn wrapper library
• kasper - a lightweight library for processing Kafka topics
• and others

Kafka Streams Libraries

• Scala
• faust - Python Stream Processing
• Southpaw - Streaming left joins for data change capture

KSQL

• KSQL JDBC Driver
• KSQL Addon - User Defined Function (UDF) for Machine Learning

Connectors

• JDBC
• Elasticsearch
• HDFS
• Beast - Kafka to BigQuery Sink
• BigQuery
• SQL
• GCloud pubsub
• BigTable
• Google Cloud Storage (GCS)
• HBase
• FileSystem
• Streaming files from a local filesystem
• kafka-connect-protobuf-converter
• IBM MQ Source
• IBM MQ Sink
• Azure IoT Hub
• Pulsar
• HTTP

Producers

• Maxwell's daemon - a mysql-to-json kafka producer

Consumers

• Spark
• Storm
• Mapreduce
• Divolte
• Azure Data Explorer/Kusto
• Firehose - Firehose is an extensible, no-code, and cloud-native service to load real-time streaming data from Kafka to data stores, data lakes, and analytical storage systems.
• Raccoon - Raccoon is a high-throughput, low-latency service to collect events in real-time from your web, mobile apps, and services using multiple network protocols and publish to Kafka.

Transformations

• Dagger - Dagger is an easy-to-use, SQL framework to process built on top of Apache Flink for stateful processing of Kafka data.
• Single Message Transformations (SMT)

Testing

• Mocked Streams - Topology testing library for Kafka Streams (no broker required)
• Unit-testing for embedded kafka services
• Kafka JUnit
• Kafka Unit
• Kafka for JUnit
• embedded-kafka - in-memory Kafka instance to run your tests against
• Kafka Containers
• Mockintosh - through asynchronous Kafka actors

Operations

CI/CD

• kafka-gitops

Operational Utilities

• Cruise control
• DoctorKafka and its successor Orion
• Brucke - Inter-cluster bridge of kafka topics
• Kafka-Kit
• Secor - service for persisting Kafka logs
• kafkabalancer
• Yelp Kafka-Utils
• Cerner Common Kafka utilities
• connectctl - manage kafka connect connectors easily

Monitoring

• Kafka Offset Exporter
• Fluent plugin
• Burrow - Kafka Consumer Lag Checking
• Remora
• Kafka Offset Monitoring
• kafka-eagle
• Kafka-prometheus
• Consumer lag monitoring

Performance tools

• sangrenel
• KLoadGen - load generator plugin for jmeter

Security

• Kafka Security Manager
• Kafka security playbook

Audit

• chaperone - A Kafka audit system

Mirroring

• MirrorMaker
• Message Handler For MirrorMaker to change topic names
• uReplicator
• MirrorTool for Kafka Connect
• Mirus

Backup

• Backup and Restore topics & offsets

Tools

• Karapace - Your Kafka essentials in one tool
• REST-proxy
• Kafka-Pixy - gRPC/REST Proxy
• Kafka Proxy providing SASL/PLAIN authentication and SSL
• Reactive/Akka API
• kafkat
• kt
• kafka-connect-tools
• Node Kafka connect
• Kafka-monitor
• Kafka Auto completion
• Kafka Cheet Sheet
• Kafka Write Protocol encoder/decoder
• Winton Kafka Streams
• Spring-kafka
• Maven quick start for Kafka Connect connectors
• Trifecta
• Kafka Specs
• Strimzi Kafka CLI

Metadata Management

• Go library for Kafka metadata in Zookeeper
• Meteor - Meteor is a scalable, easy-to-use, extensible metadata collection framework from the different cloud providers and on-prem sources and publish to Kafka.

Schema Management

• Stencil - Stencil is a Protobuf schema registry for Kafka that enables you to create, maintain and consume protobuf messages and APIs dynamically, efficiently, and reliably.
• Schema registry
• Node Schema registry
• Schema Registry Transfer SMT
• Expedia Stream Registry

UI

• kafka-manager
• Kafka HQ
• Kafdrop 3.x – web UI for Kafka. For versions older than 0.10.0 use Kafdrop 2.x.
• Kafka Webview
• Kafka Topics UI
• Kafka Connect UI
• Kafka Schema UI
• Kafkawize - self service topic management portal
• Kadmin - Kafka Producer/Consumer UI
• Kowl
• Kafka Streams Topology Sketch Diagram Visualization
• Web based GUI

Deployment

Kubernetes

• Kafka k8s operator
• Strimzi Operator for deploying and running Apache Kafka on Kubernetes
• konsumerator - Kafka Consumer Operator

Docker

• fast-data-dev

Integrations

• Ecosystem
• hermes
• IBMStreams
• Brooklin

Alternative Implementations

• Go
• C/C++

Projects built on top of Kafka

• In-Memory Cache Backed by Kafka
• ransactional Metadata Store Backed by Kafka

Unsorted

• https://github.com/Landoop/kafka-connect-kcql-smt
• https://github.com/nodefluent/purpur
• Azkarra Streams - lightweight java framework which makes easy to create production-ready Kafka Streams applications