pykafka
pykafka copied to clipboard
degraded consumer operation when partitions have unrecoverable errors
Consider a mode of operation for the consumer in which some partitions being inaccessible (possibly due to GroupLoadInProgressCode errors) does not cause the consumer to crash. fetch_offsets in particular doesn't return successfully unless all partitions provide offset information without error. It's conceivable that the consumer could consume only from the partitions it can successfully fetch offset information for, while ignoring the others.
Anecdotally, the Java client has the same issue as pykafka in this respect.