kafka-message-scheduler
kafka-message-scheduler copied to clipboard
sky-uk
Support horizontal scaling
When the kafka-topic-loader supports loading a given partition instead of an entire topic we will be able to spawn multiple instances of the KMS and let Kafka assign partitions to each node.
We'll need to use the source per partition from alpakka Kafka and feed the partitions it emits to the topic loader
This will need to be heavily tested, whoever picks this up can asses either sbt docker compose or akkas multi jvm testkit for that testing. This app doesn't require an akka cluster so sbt docker compose is probably what we want? I believe you can scale up and down instances with it? if not multi jvm testkit maybe.
We need to test using a topic with multiple partitions and instances. For example a topic with 12 partitions and 4 kms instances
the kms is composed of 2 streams, the reader stream and the publisher stream, reader stream reads schedules from kafka and writes them to the scheduling actor which does the actual scheduling. When an event fires it goes to the publisher stream.
We only need one publisher stream per jvm, we'll create a reader stream per kafka partition so the main changes will be here.
We'll have to add some cleanup for when rebalancing happens, so when a reader stream completes (completes, not fails) because of partition reassignment we'll just want to kill the scheduling actor for that reader stream by sending a poison pill to the actor or something. A problem with this will be that the terminator kills the app when the scheduling or publisher actor dies. So we'll have to not do that in the case of when we cleanly shutdown the scheduling actor when reassignment happens
The topic loader now supports loading from specified partitions (will link PR later)
It doesn't properly. We need to do https://github.com/sky-uk/kafka-topic-loader/issues/32 first