Dominic Evans

Note: there is likely still discussion and changes needed here, but I wanted to open a PR to re-open discussions as the old one had long been abandoned and closed.

As per https://github.com/Shopify/sarama/issues/2115#issuecomment-1018360997 > Yes looks likely to be due to the change in behaviour in Kafka 2.8.0 onward [KAFKA-9751](https://issues.apache.org/jira/browse/KAFKA-9751) whereby the broker is asynchronously forwarding the auto create of...

Hmm, having investigated this some more it seems like a change in behaviour on the broker in 2.8.x onward means that __consumer_offsets is not getting created by a MetadataRequest alone...

Kafka dev mailing list agree this is a regression in the broker. https://lists.apache.org/thread/cxzq6009t2vsyc05tcro77mov12zhpvm Raised upstream as https://issues.apache.org/jira/browse/KAFKA-13612 The only workaround for Burrow would be to actively join a consumer group...

@Jacob-bzx thanks for spotting and debugging this problem I can see that the mutex approach will solve the data race, but I'm less sure about the potential panic. I wonder...

@wricardo thanks for reporting this. I've applied the fix, but I wasn't able to find a linter that would prevent us from (re-)introducing this problem in the future, as none...

@shweta-fourkites as you may have seen, functionally the Sync producer is just a wrapper around the Async producer https://github.com/Shopify/sarama/blob/619d4ef4d2061b15c438d1e9dfcb3bf5390493d4/sync_producer.go#L60-L76 so certainly compression is equally applicable — it is applied to...

@shweta-fourkites it depends what you mean by "kafka endpoint". All batching is done on a per-broker level and which broker you send messages is arranged on a per-partition level. So...

You're welcome :)

Closing as answered