kafka_ex

kafka_ex copied to clipboard

Call for comments - Should ConsumerGroup support a back pressure mechanism and how?

1

What I mean here by back pressure is dwelling between `fetch` calls. This is an important part of many production systems where a stream processor may need to pause consuming...

dantswain

Idea: Document snappy usage on server

1

Hi there, we are currently exploring `kafka_ex` in combination with `snappy`. It works quite nice, but we hit an issue when we deployed to a staging system. It was crashing...

namxam

KafkaEx.stream timing out trying to fetch more messages even with no_wait_at_logend

1

## Issue description After banging my head trying to understand why I was getting timeouts from Kafka I noticed that the timeout was on a fetch call where the `offset`...

yogodoshi

Support current kafka compression types

3

Currently kafka supports more compression types by default than kafka_ex which can cause a client to fail when attempting to read such a topic. In particular confluent platform uses some...

technomage

A down broker causes unnecessary restarts

1

If we have a three broker cluster in the config, but one of those brokers is down, the application appears to continually crash and restart. The cluster is set up...

dantswain

Test with dockerized kafka/zookeeper

2

This will open the door to a few things: * Testing against multiple versions of kafka * Ensuring a clean slate for every run of the tests - maybe eventually...

dantswain

Fixes #291 Seems like there still was some binary concatenation in the protocol modules as far as I could tell, so I followed up on the work done in #290...

mjparrott

Producer doesn't handle `not_leader_for_partition` errors

1

If the connection to a partition leader is broken, KafkaEx handles it by triggering a metadata request and updating its cached metadata. It is possible that the leader can change...

shamilpd

Error on fetching message sets larger than max_bytes

6

See #268 for details and some discussion. > We had a recent production issue where our consumer was unable to progress. It turns out that our Kafka broker was sending...

dantswain

Use iolists when building packets

9

#290 / #231 Implemented using iolists / iodata when building produce protocol messages. This results in a significant speedup in those parts of the code, especially with larger messages/message sets....

joshuawscott