rdkafka-ruby icon indicating copy to clipboard operation
rdkafka-ruby copied to clipboard

Support for String partitioner config

Open mollyegibson opened this issue 2 years ago • 0 comments

I am using racecar version 2.8.2, rdkafka 0.12.0, and kafka 3.2.1. I am producing messages to the same topic as a Java service, and I need to set the producer partitioner_name to murmur2 so that they hash the partition keys in the same way and events with the same partition key are sent to the same partition. I have set my config in racecar.yaml to include the partitioner like this:

# These config values will be shared by all environments but can be overridden.
common: &common
  client_id: "my-client-id"
  producer:
    - partitioner=murmur2

Racecar is then pulling that partitioner value into a hash and configuring an rdkafka producer: racecar/lib/racecar/config.rb

however, when rdkafka is configuring the consumer, it expects the partitioner configuration to be a symbol rather than a String: https://github.com/appsignal/rdkafka-ruby/blob/c02f217a189b71e3e253fe4ad59286b5fc9d4034/lib/rdkafka/config.rb#L184

I've raised a PR in rdkafka to support both string and symbol keys for the partitioner config: https://github.com/appsignal/rdkafka-ruby/pull/213

mollyegibson avatar Aug 25 '22 23:08 mollyegibson