how to configure oauthbearer token refresh

[bruce-szalwinski-he]

I'm trying to use rdkafka-ruby with AWS MSK. For authentication, I would like to use sasl_ssl with the sasl mechanism being OAUTHBEARER. It looks like support for AWS MSK IAM Auth was added to ruby-kafka. Is such support available in rdkafka as well? If so, I'm missing a step on how to configure it.

My config map looks like:

kafka_config = {
  "bootstrap.servers": bs,
  "security.protocol": "sasl_ssl",
  "sasl.mechanisms": "OAUTHBEARER",
  "oauthbearer_token_refresh_cb": MskTokenProvider.new,
  "client.id": Socket.gethostname,
}

The MskTokenProvider looks like

class MskTokenProvider
  def token
    generate_auth_token(region, role_arn)
  end
end

Starting the producer with

    def self.start!(kafka_config)
      puts "#{PREFIX} booting"
      @producer = Rdkafka::Config.new(kafka_config).producer
    end

results in

/usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:263:in `block (2 levels) in native_config': Property "oauthbearer_token_refresh_cb" must be set through dedicated .._set_..() function (Rdkafka::Config::ConfigError)
	from /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:253:in `each'
	from /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:253:in `block in native_config'
	from <internal:kernel>:90:in `tap'
	from /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:251:in `native_config'
	from /usr/local/bundle/gems/rdkafka-0.15.1/lib/rdkafka/config.rb:201:in `producer'
	from /usr/src/app/lib/producer.rb:11:in `start!'
	from /usr/src/app/kafka.rb:35:in `<main>'

[bruce-szalwinski-he]

This comment from the AWS folks say they have no plans to release signer library in Ruby. So I guess all languages doesn't include Ruby.

[bruce-szalwinski-he]

I also see someone from ancient times attempting to write a custom oauthbearer_token_refresh_cb.

[mensfeld]

Is such support available in rdkafka as well?

Not currently.

Here is my statement about this: https://github.com/karafka/karafka/issues/1767

TL;DR: I do not plan to work on it anytime soon but I do have it in my plans. Appropriate on-init refresh callbacks for Waterdrop and Karafka have already been added, but I cannot commit to the bindings layer in Q1 and probably Q2 because of the reasons mentioned in the above thread.

I will keep this issue open as a reference point.

[mensfeld]

I added the "help wanted" in case anyone is willing to add this as a PR :) if not, as stated, it will be done at some point.

[bruce-szalwinski-he]

Took a stab at this in https://github.com/karafka/rdkafka-ruby/pull/410