kafka-github-actions

kafka-github-actions copied to clipboard

README

• Overview
• Instructions
• Resources

Overview

This demo uses GitHub Actions for developing and valiating Apache Kafka client application, focusing on a Python producer and consumer using Kafka Python Client, plus examples for testing schema compatibility using both Python and Java.

The code defines an GitHub Actions workflow kafka-app with jobs to demonstrate a few capabilities:

• build: Build a Docker image for the Kafka Python client application
• sanity: Sanity test the app with rdkafka mock
• local: Locally test the app using an action called cp-all-in-one-action to run Confluent Platform from a Docker Compose file
• local-avro: Locally test the app with Schema Registry and Avro using an action called cp-all-in-one-action to run Confluent Platform from a Docker Compose file
• ccloud: Test the app to Confluent Cloud
• ccloud-avro: Test the app to Confluent Cloud with Confluent Cloud Schema Registry and Avro
• ccloud-python-schema-compatibility: Use SchemaRegistryClient.test_compatibility() as shown in src/test_schema_compatibility.py to check a new schema's compatibility to the existing schema in Confluent Cloud Schema Registry (Python)
• ccloud-java-schema-compatibility: Use kafka-schema-registry-maven-plugin as shown in the pom.xml to check a new schema's compatibility to the existing schema in Confluent Cloud Schema Registry (Java)

Instructions

1. Provision a Kafka cluster, enable Schema Registry and create an API key and secret on Confluent Cloud. Use the quickstart, Confluent Terraform Provider, or ccloud-stack.

2. In your GitHub account, create Encrypted Secrets for your Confluent Cloud cluster and credentials:

# Confluent Cloud Kafka cluster
CONFLUENT_BOOTSTRAP_SERVERS
CONFLUENT_API_KEY
CONFLUENT_API_SECRET
# Confluent Cloud Schema Registry
CONFLUENT_SCHEMA_REGISTRY_URL
CONFLUENT_BASIC_AUTH_USER_INFO

3. Fork this repository and push a change to trigger the workflow.

Resources

• Blog: more details on Apache Kafka CI/CD with GitHub Actions
• Confluent Cloud: Cloud-native service for Apache Kafka
• GitHub Actions for ksqlDB: demonstrates schema migrations upon a ksqlDB cluster running in Confluent Cloud
• Kafka Python Client
• GitHub Actions