Passing sasl.jaas.config property seems impossible when starting apicurio-registry-storage-kafka-1.2.3.Final with Docker

[elakito]

This is the follow up question raised in https://github.com/Apicurio/apicurio-registry/issues/10#issuecomment-657489360

It is still not clear to me if it is possible to run apicurio-registry-storage-kafka-1.2.3.Final in docker and setting sasl.jaas.config over the JAVA_OPTIONS environment variable.

I tried several variants.

1. First, a smoke test without passing JAVA_OPTIONS.

$ docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' apicurio/apicurio-registry-kafka:1.2.3.Final
exec java -Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar

__  ____  __  _____   ___  __ ____  ______ 
 --/ __ \/ / / / _ | / _ \/ //_/ / / / __/ 
 -/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \   
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/   
2020-07-31 12:38:47,448 WARN  [io.qua.config] (main) Unrecognized configuration key "quarkus.datasource.username" was provided; it will be ignored
...
	sasl.mechanism = GSSAPI
	security.protocol = PLAINTEXT
...

2. Setting JAVA_OPTIONS but omitting the required sasl.jaas.config property.

$ docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN' apicurio/apicurio-registry-kafka:1.2.3.Final

exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
...
	sasl.mechanism = PLAIN
	security.protocol = SASL_PLAINTEXT
...
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:820)
	at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
	at io.apicurio.registry.utils.kafka.ConsumerContainer.consumerLoop(ConsumerContainer.java:143)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
...

One can see the container is started but complains about the missing jaas.config as expected.

3. Setting JAVA_OPTIONS with the jaas.config (variant-1 using 'JAVA_OPTIONS=... ...sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret"')

docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret"' apicurio/apicurio-registry-kafka:1.2.3.Final

exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required

It seems it didn't even parse the properties correctly.

4. Setting JAVA_OPTIONS with the jaas.config (variant-2 'JAVA_OPTIONS=... ...sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret";"' )

docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";"' apicurio/apicurio-registry-kafka:1.2.3.Final

exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required

the same error.

5 Setting JAVA_OPTIONS with the jaas.config (variant-3 "JAVA_OPTIONS=... ...sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\"")

docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e "JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\"" apicurio/apicurio-registry-kafka:1.2.3.Final

exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required

the same error

Could someone tell me if it not possible or if it is possible how I can start the docker container?

[EricWittmann]

@alesj or @famartinrh - can either of you guys shed any light on this?

[EricWittmann]

Actually @jsenko might know the answer here!

[forsberg]

Perhaps one way forward is to generate a configuration file with the required values, then map them into the docker container as config/application.properties? This should, according to Quarkus Documentation override the default properties.

You could even insert e.g. the password from an environment variable using Environment Variable expansion

Untested. though, just guessing based on source code and docs.

[EricWittmann]

I think the latest version of Quarkus has some new property handling features that we'll be able to take advantage of in the next release (we've already made the necessary changes but haven't done a release since then). I don't recall the technical specifics but @alesj would know. I'm hoping to do a release soon (after merging all of our 1.2.x changes) but am being thwarted by the continuing aftermath of the hurricane from last week. :(

[alesj]

Yeah, Quarkus has that - config/application.properties -- just not the version we're using in 1.2.x With Quarkus upgrade you should be able to use that ootb. Till then, JAVA_OPTIONS env var is your best option:

• https://github.com/Apicurio/apicurio-registry#security

[EricWittmann]

The next release (1.3.0.Final) will include a newer version of Quarkus. I'm hoping to have that release done this week. I've merged the changes from the 1.2.x branch and am hoping to also get Carles' new client merged as well prior to the release.

[elakito]

@EricWittmann @alesj Thanks for the information. I thought about trying to make apicurio take those non trivial java system properties from the dedicated env variables instead of using a single JAVA_OPTIONS env variable. If 1.3 will provide a file based configuration, that will be much better and flexible. That sounds good.

[forsberg]

I did manage to get this to work by mapping in an config/application.properties from a Configmap into the Docker container's filesystem using Kubernetes, so I think the version of quarkus in use actually do support this.

[PlugaruT]

@forsberg How did you managed to use the config/application.properties file? Where should be located in the container? I'm also trying to do that but with no luck, see #1805

[forsberg]

@PlugaruT - this was a long time ago, and I'm not actively using Apicurio anymore, but digging into my chart it looks like I have a configmap with the application.properties as key, and I mount it under /deployments/config, but that's probably as much information I have for you.

Reading Quarkus Configuration Reference this seems valid if we assume that /deployments is the working directory where the registry is started (see point 4 in the list of config sources), and if I shell into the pod I actually had running, this is the case.

So, in essence, mount your application.properties so that it becomes $PWD/conf/application.properties and Quarkus will read it.

[PlugaruT]

@forsberg Thanks a lot, it seems to work if I mount it under /deployments. Right now, if I exec into the container, the $PWD is pointing to /home/jboss and I was mounting it there...

[forsberg]

@PlugaruT - glad you made it work! :+1:

As a general observation, sometimes having a look at /proc/<pid> to see where /proc/<pid>/pwd points to can be enlightening when it comes to the real working directory of a program.

[EricWittmann]

Note to @jsenko and @carlesarnal

[TomBillietKlarrio]

Took me a while to figure out, but setting these environment variables get you going (sample for AWS MSK)

REGISTRY_KAFKA_COMMON_SECURITY_PROTOCOL: SASL_SSL
REGISTRY_KAFKA_COMMON_SASL_JAAS_CONFIG: "software.amazon.msk.auth.iam.IAMLoginModule required;"
REGISTRY_KAFKA_COMMON_SASL_MECHANISM: AWS_MSK_IAM
REGISTRY_KAFKA_COMMON_SASL_CLIENT_CALLBACK_HANDLER_CLASS: software.amazon.msk.auth.iam.IAMClientCallbackHandler

[jsenko]

We've discussed this issue and decided on supporting management of environment variables in the Apicurio CR like so:

spec:
 configuration:
   persistence: kafkasql
   kafkasql:
     bootstrapServers: "my-cluster-kafka-bootstrap.registry-example-kafkasql-scram.svc:9093"
 env:
    - key: REGISTRY_KAFKA_COMMON_SECURITY_PROTOCOL
      value: SASL_SSL
    - key: REGISTRY_KAFKA_COMMON_SASL_JAAS_CONFIG
      value: "software.amazon.msk.auth.iam.IAMLoginModule required;"
    - key: REGISTRY_KAFKA_COMMON_SASL_MECHANISM
      value: AWS_MSK_IAM
    - key: REGISTRY_KAFKA_COMMON_SASL_CLIENT_CALLBACK_HANDLER_CLASS
      value: software.amazon.msk.auth.iam.IAMClientCallbackHandler

Solving https://github.com/Apicurio/apicurio-registry-operator/issues/167 and adding documentation about available env. variables.