decaton
decaton copied to clipboard
line
High throughput asynchronous task processing on Apache Kafka
- Currently, Decaton's SubPartitioner uses murmur2 hash of the key to determine the subpartition to ensure per-key processing ordering guarantee. - However, sometimes we want to use round-robin strategy even...
After [this change](https://github.com/line/decaton/pull/99/files#diff-4df160cb6c95ec9c8815d2af1dda8a9ebe841c74c4e094bd3ef8e75074dd0af5L121) in https://github.com/line/decaton/pull/99, decaton stopped incrementing `taskMetrics.tasksError` for synchronous failure. This PR wants to fix `tasksError` to work again. I applied quick patch as first PR, but implementing...
- Kafka 2.4.0 introduces `cooperative rebalancing protocol`, which mitigates the stop-the-world effect upon rebalancing * https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol - Users can opt-in cooperative rebalancing by specifying `CooperativeStickyAssignor` in `partition.assignment.strategy` config or implementing...
- In Kafka, it's common to use some entity's identifier (e.g. user-id) to ensure task processing ordering per entity. - At the same time, this tends to cause "hot-spot" key...
We can inject a custom `org.apache.kafka.clients.producer.Partitioner` by implementing it and passing it through `DecatonClientBuilder#producerConfig`. However, A decaton client wraps a task set by our application in `DecatonTaskRequest`. So, to determine...
# Motivation - To add a documentation for `BathingProcessor` like [task-compaction.adoc](https://github.com/line/decaton/blob/master/docs/task-compaction.adoc). Related Issue: https://github.com/line/decaton/issues/128 Related PR: https://github.com/line/decaton/pull/139
- Task batching is a common-pattern that many Decaton users often implement by their own * i.e. Batching several tasks of type `T` to `List` and process them at once....
It would be useful for precise detection and spot the causing record when `DeferredCompletion` leak occurred by user-supplied code.
We are consuming Kafka topic using Decaton processor with retrying. However, `ProcessorsBuilder.consuming(String topic, TaskExtractor taskExtractor)` is not working correctly with `DefaultTaskExtractor`. `retryTaskExtractor` will unwrap `DecatonTaskRequest` using `DefaultTaskExtractor`, then `taskExtractor.extract()` here....
## Summary - Currently, Decaton uses `KafkaConsumer#poll(long)` which is deprecated and could be removed in future Kafka release