Event-Driven microservices with Kafka Streams

A skeleton example of how to use Kafka Streams with Event-Driven microservices.

This is an application that generates a simple football statistics like match scores, team and player rankings. It should be a little more comprehensible than a typical code examples based on orders, payments etc.

You can think of this application as a supplement to the basic code examples of Kafka Streams. Kafka Streams is used here extensively for many purposes like data processing, building materialized views, persisting the domain model and exposing the output of services. Furthermore, there is no traditional database, local state stores instead (PostgreSQL plays a role of legacy system only).

This codebase is trying to apply the idea of stateful streams processing with Kafka and Kafka Streams. These blog posts are a great introduction to that concept:

• The Data Dichotomy: Rethinking the Way We Treat Data and Services
• Build Services on a Backbone of Events
• Using Apache Kafka as a Scalable, Event-Driven Backbone for Service Architectures
• Event sourcing using Kafka

Services

• kafka,
• zookeeper,
• postgres - in the role of an external data source,
• football-match - REST requests transformation into events, input validation with the domain model,
• connect - Kafka Connect with Debezium PostgreSQL connector, writes events for insert or update operations on PLAYERS table to a single Kafka topic,
• football-player is receiving notifications from connect service and creating only a single PlayerStartedCareer event using Processor API (see the code),
• football-view-basic - builds the statistics from the events using Kafka Streams DSL, it's the main place where streams are created and processed (see below),
• football-view-top - builds the Top 10 Players ranking based on the statistics generated by football-view-basic and not on the events,
• football-ui - saves the statistics into materialized views (local state stores), exposes them via REST API and publishes using WebSocket.

Additional modules:

• football-common - contains some code that is shared among microservices (obviously it's not a good practice for production), especially:
  • events are used by football-match, football-player and football_view,
  • statistics - football-view-basic, football-view-top, football-ui,
• football-tests - demo application and integration tests.

Events and Streams

Each event represents a state change that occurred to the system.

These events are the source for stream processing. For example, in order to determine the result of a match, you must join MatchStarted and GoalScored streams and then count the goals (see the code).

Domain Model

Kafka Topics

Topics are organized under logical groups as follows:

• fb-event.* - containing events generated by football-match and football-player,
• fb-view.* - statistics produced by football-view-basic and football-view-top,
• fb-connect.* - events from the PostgreSQL connector.

The prefixes above aim to separate the application topics from the internal Kafka Streams and Debezium topics.

REST Endpoints

There are only two REST endpoits and both of them are accessed from the outside of the system:

• Query interface in football-ui,
• Command interface in football-match.

How to run

Requirements: Java 10, Docker with at least 6 GB memory limit.

1. Add kafka bootstrap address to your /etc/hosts:

   127.0.0.1 kafka
   127.0.0.1 postgres
   127.0.0.1 connect
   127.0.0.1 football-match
   127.0.0.1 football-player
   127.0.0.1 football-view-basic
   127.0.0.1 football-view-top
   127.0.0.1 football-ui
   

2. Build microservices and Docker containers:

   cd football
   mvn install
   

3. Build and run the demo application:

   cd football-tests
   mvn install -DskipTests
   java -jar target/football-tests-0.0.1-SNAPSHOT-jar-with-dependencies.jar
   

   It will start Docker containers and wait until all microservices are ready.

   If the default startup timeout (180 sec) will be insufficient, try to increase it:

   java -jar target/football-tests-0.0.1-SNAPSHOT-jar-with-dependencies.jar 400
   

4. Wait a minute and launch http://football-ui:18080/. In a few seconds you should see some statistics updated in real time:

The input is historical data from English Football League Championship. The historical events (goal scored, card received etc.) are played back every few seconds.