Clarify documentation on inter broker configuration on SASL_PLAINTEXT?

[DeepRed]

I am new to both KeyCloak and Kafka but currently working on using your OAuth2 lib. I have read your documentation numerous times and looked into Kafka and Zookeeper documentation but have a hard time figuring out how to go from PLAINTEXT to SASL_PLAINTEXT with OAUTHBEARER on inter broker communication. Reading your documentation leads me to believe that this should be possible. If I have understood it correctly going to SASL_PLAINTEXT on inter broker communication would require me to configure the JAAS config. When I make the switch to SASL_PLAINTEXT in my Helm Chart from Confuent, then Kafka lets me know that KAFKA_OPTS is a required environment variable and it needs to contain JAAS config. Looking at the JAAS config it need to implement userid and password for Zookeeper integration. As I understand it configuring userid and password for Zookeeper requires a SSL certificate.

Am I correct in my assumptions above? If a SSL certificate is required then I might as well go straight to SASL_SSL with OAUTHBEARER for external and PLAINTEXT for inter broker communication. Is this a correct conclusion? Or is there a way to go to SASL_PLAINTEXT without the need of a certificate when communicating with Zookeeper? Could you comment on how to go about this and what the necessary configuration is? A simple list of a few steps would help me alot.

My config currently includes:

listener.security.protocol.map=PLAINTEXT:PLAINTEXT,EXTERNAL:SASL_PLAINTEXT
sasl.enabled.mechanisms=OAUTHBEARER,PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN
security.inter.broker.protocol=PLAINTEXT
super.users=User:service-account-kafka;User:ANONYMOUS

Running inter broker communication as PLAINTEXT requires the anonymous to become a super user. This is visible in the logs.

DEBUG 	Authorization GRANTED - 
	user is a superuser: User:ANONYMOUS, 
	operation: Read, 
	resource: Topic:LITERAL:kafka-cp-kafka-connect-offset (io.strimzi.kafka.oauth.server.authorizer.KeycloakRBACAuthorizer.grant)

But integrating externally uses the OAuth bearer token and the user is authenticated and authorized, which also is visible in the logs.

DEBUG 	Authorization grants for user OAuthKafkaPrincipal(User:user-account-x, session:775475596, token:******): [
	{"scopes":["Read","Write","Describe"],"rsid":"fa5f56f4*****","rsname":"Topic:some-topic-name"},
	{"scopes":["Read","Describe"],"rsid":"bfe55748******","rsname":"Group:*"}] io.strimzi.kafka.oauth.server.authorizer.KeycloakRBACAuthorizer) 
DEBUG 	Authorization GRANTED - 
	cluster: kafka-cluster, 
	user: OAuthKafkaPrincipal(User:user-account-x, session:775475596, token:******), 
	operation: Describe, resource: Topic:LITERAL:some-topic-name
	Granted scopes for resource (Topic:some-topic-name:): [Read, Write, Describe] (io.strimzi.kafka.oauth.server.authorizer.KeycloakRBACAuthorizer.grant) 

Of course running ANONYMOUS as super user is somehting that cannot be done in a production environment. Even in the test environment this feels wrong and that is why I wish to go from PLAINTEXT to SASL_PLAINTEXT. But if it is easier to configure SASL_SSL with a certificate then maybe I should maintain certificates for the test environment as well.

And a huge thank you for your repo and documentation. It seems the authorization works really good in conjunction with KeyCloak.

[scholzj]

Communication between Kafka and Zookeeper is completely separate - it has its own configuration and its own mechanisms (and this lib does not really deal with it). Also the user used for inter broker communication would most probably need a lot of ACLs ... so normally making it super user makes a lot of sense anyway.

@mstruk Do you have some example of using OAUth for replication? Did you ever tried it?

[mstruk]

It's complex, that is why you may find it poorly documented. We document strimzi-kafka-oauth topics, and try not to repeat documenting Apache Kafka topics. Any suggestions for improvements are very welcome.

First, some initial observation:

PLAINTEXT security protocol means NO authentication over unencrypted connection. SASL_PLAINTEXT means authentication over unencrypted connection.

Analogously ...

SSL means NO authentication over encrypted TLS connection unless you specify 'ssl.client.auth=required' for the listener in which case mutual TLS (mTLS) authentication is required. SASL_SSL mean authentication over encrypted TLS connection.

SASL configuration only kicks in when one of SASL_* security protocols is configured for the listener.

In order to use OAuth for inter-broker connections you have to specify the following two setting:

• sasl.mechanism.inter.broker.protocol
• inter.broker.listener.name

You also need to configure the listener with SASL_PLAINTEXT (rather than PLAINTEXT) or SASL_SSL (rather than SSL). For proper security you should use SASL_SSL.

Then, you also need to configure the sasl.jaas.config with client configuration options, not just the server configuration options.

Now, to make matters more complicated, when you configure your listener to support OAuth, you can configure it in such a way that your clients have to use SASL_OAUTHBEARER, but you can also configure it to support what we call OAuth over PLAIN where the client actually uses SASL_PLAIN protocol and sends clientId as 'username', and client secret as 'password'. This mechanism is primarily for clients that do not support SASL_OAUTHBEARER. It does not really make much sense on the broker for inter-broker communication because SASL_OAUTHBEARER is supported, but it makes sense for your external clients. Keep in mind that simply enabling the PLAIN sasl mechanism will still result in using the configured sasl.jaas.config login module which for OAuth over PLAIN enabled listener will not be PlainLoginModule but io.strimzi.kafka.oauth.server.plain.JaasServerOauthOverPlainValidatorCallbackHandler which you can not use for inter-broker connectivity using PLAIN. In short you can use OAUTHBEARER, but you can't use PLAIN (as in OAuth over PLAIN) for inter-broker communication.

Continuing with your specific example ... It would be more meaningful to call your inter-broker listener INTERBROKER or REPLICATION rather than PLAINTEXT. You should use OAUTHBEARER for inter-broker communication rather than PLAIN.

Thus your config should look something like the following (with possible typos and mistakes, and ignoring that for production you should use SASL_SSL, and also access your sso server using https):

listener.security.protocol.map=REPLICATION:SASL_PLAINTEXT,EXTERNAL:SASL_PLAINTEXT
sasl.enabled.mechanisms=OAUTHBEARER,PLAIN
sasl.mechanism.inter.broker.protocol=OAUTHBEARER
inter.broker.listener.name=REPLICATION

# Because REPLICATION listener is used for inter-broker communication it also requires the 'client-side' login callback handler and corresponding configuration:

listener.name.replication.oauthbearer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
    oauth.client.id="kafka" \
    oauth.client.secret="kafka-secret" \
    oauth.token.endpoint.uri="http://sso:8080/auth/realms/demo/protocol/openid-connect/token" \
    oauth.valid.issuer.uri="http://sso:8080/auth/realms/demo" \
    oauth.jwks.endpoint.uri="http://sso:8080/auth/realms/demo/protocol/openid-connect/certs" \
    oauth.username.claim="preferred_username" ;

# Server-side-authentication handler
listener.name.replication.oauthbearer.sasl.server.callback.handler.class=io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler

# Login-as-a-client handler
listener.name.replication.oauthbearer.sasl.login.callback.handler.class=io.strimzi.kafka.oauth.client.JaasClientOauthLoginCallbackHandler


# The EXTERNAL listener only needs server-side-authentication support because we don't use it for inter-broker configuration:

listener.name.external.oauthbearer.sasl.jaas.config=org.apache.kafka.common.security.oauthbearer.OAuthBearerLoginModule required \
    oauth.valid.issuer.uri="http://sso:8080/auth/realms/demo" \
    oauth.jwks.endpoint.uri="http://sso:8080/auth/realms/demo/protocol/openid-connect/certs" \
    oauth.username.claim="preferred_username" \
    unsecuredLoginStringClaim_sub="unused" ;

# The last parameter is needed for configuration to pass OAuthBearerLoginModule validation when we don't specify a custom sasl.login.callback.handler.class

# Server-side-authentication handler
listener.name.external.oauthbearer.sasl.server.callback.handler.class=io.strimzi.kafka.oauth.server.JaasServerOauthValidatorCallbackHandler


# On EXTERNAL listener we may also want to support OAuth over PLAIN
listener.name.external.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required \
    oauth.token.endpoint.uri="http://sso:8080/auth/realms/demo/protocol/openid-connect/token" \
    oauth.valid.issuer.uri="http://sso:8080/auth/realms/demo" \
    oauth.jwks.endpoint.uri="http://sso:8080/auth/realms/demo/protocol/openid-connect/certs" \
    oauth.username.claim="preferred_username" \
    unsecuredLoginStringClaim_sub="unused" ;

# Server-side-authentication handler
listener.name.external.plain.sasl.server.callback.handler.class=io.strimzi.kafka.oauth.server.plain.JaasServerOauthOverPlainValidatorCallbackHandler

This is without authorizer configuration.

It's quite complicated as you can see. In Strimzi Operator we use mTLS to configure inter-broker authentication, and we set up all the certificates for you. Makes things much easier to get up and running.

[mstruk]

@mstruk Do you have some example of using OAUth for replication? Did you ever tried it?

Many of our examples and testsuite tests use OAUTHBEARER for inter-broker communication. Single listener configurations, for example, have no other option (as we configure them with OAUTHBEARER).

[DeepRed]

Excellent answer, thank you @scholzj and @mstruk. I had previously tried some, maybe even most, of the configuration in your example but must have missed some details. I followed your suggestions and now my inter broker communication is running fine with SASL_PLAINTEXT. Thanks alot! Maybe that answer is a candidate for the documentation? Now we will run some evaluation in our development environment.