apicurio-registry
apicurio-registry copied to clipboard
Passing sasl.jaas.config property seems impossible when starting apicurio-registry-storage-kafka-1.2.3.Final with Docker
This is the follow up question raised in https://github.com/Apicurio/apicurio-registry/issues/10#issuecomment-657489360
It is still not clear to me if it is possible to run apicurio-registry-storage-kafka-1.2.3.Final in docker and setting sasl.jaas.config over the JAVA_OPTIONS environment variable.
I tried several variants.
- First, a smoke test without passing JAVA_OPTIONS.
$ docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' apicurio/apicurio-registry-kafka:1.2.3.Final
exec java -Dquarkus.http.host=0.0.0.0 -Djava.util.logging.manager=org.jboss.logmanager.LogManager -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
__ ____ __ _____ ___ __ ____ ______
--/ __ \/ / / / _ | / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ |/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ |_/_/|_/_/|_|\____/___/
2020-07-31 12:38:47,448 WARN [io.qua.config] (main) Unrecognized configuration key "quarkus.datasource.username" was provided; it will be ignored
...
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
...
- Setting JAVA_OPTIONS but omitting the required
sasl.jaas.config
property.
$ docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN' apicurio/apicurio-registry-kafka:1.2.3.Final
exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
...
sasl.mechanism = PLAIN
security.protocol = SASL_PLAINTEXT
...
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:820)
at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
at io.apicurio.registry.utils.kafka.ConsumerContainer.consumerLoop(ConsumerContainer.java:143)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalArgumentException: Could not find a 'KafkaClient' entry in the JAAS configuration. System property 'java.security.auth.login.config' is not set
...
One can see the container is started but complains about the missing jaas.config as expected.
- Setting JAVA_OPTIONS with the jaas.config (variant-1 using
'JAVA_OPTIONS=... ...sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret"'
)
docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret"' apicurio/apicurio-registry-kafka:1.2.3.Final
exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required
It seems it didn't even parse the properties correctly.
- Setting JAVA_OPTIONS with the jaas.config (variant-2
'JAVA_OPTIONS=... ...sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret";"'
)
docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e 'JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";"' apicurio/apicurio-registry-kafka:1.2.3.Final
exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=\"kafkaclient\" password=\"kafkaclient-secret\";" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required
the same error.
5 Setting JAVA_OPTIONS with the jaas.config (variant-3 "JAVA_OPTIONS=... ...sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\""
)
docker run -it -p 8080:8080 -e 'KAFKA_BOOTSTRAP_SERVERS=host.docker.internal:9092' -e 'APPLICATION_ID=my-registry-kafka' -e "JAVA_OPTIONS=-D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config=\"org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret\"" apicurio/apicurio-registry-kafka:1.2.3.Final
exec java -D%prod.registry.kafka.storage-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.storage-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.storage-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.storage-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-producer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-producer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-producer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -D%prod.registry.kafka.snapshot-consumer.security.protocol=SASL_PLAINTEXT -D%prod.registry.kafka.snapshot-consumer.sasl.mechanism=PLAIN -D%prod.registry.kafka.snapshot-consumer.sasl.jaas.config="org.apache.kafka.common.security.plain.PlainLoginModule required username=kafkaclient password=kafkaclient-secret" -javaagent:/opt/agent-bond/agent-bond.jar=jmx_exporter{{9779:/opt/agent-bond/jmx_exporter_config.yml}} -XX:+UseParallelGC -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MinHeapFreeRatio=20 -XX:MaxHeapFreeRatio=40 -XX:+ExitOnOutOfMemoryError -cp . -jar /deployments/apicurio-registry-storage-kafka-1.2.3.Final-runner.jar
Error: Could not find or load main class required
the same error
Could someone tell me if it not possible or if it is possible how I can start the docker container?
@alesj or @famartinrh - can either of you guys shed any light on this?
Actually @jsenko might know the answer here!
Perhaps one way forward is to generate a configuration file with the required values, then map them into the docker container as config/application.properties? This should, according to Quarkus Documentation override the default properties.
You could even insert e.g. the password from an environment variable using Environment Variable expansion
Untested. though, just guessing based on source code and docs.
I think the latest version of Quarkus has some new property handling features that we'll be able to take advantage of in the next release (we've already made the necessary changes but haven't done a release since then). I don't recall the technical specifics but @alesj would know. I'm hoping to do a release soon (after merging all of our 1.2.x changes) but am being thwarted by the continuing aftermath of the hurricane from last week. :(
Yeah, Quarkus has that - config/application.properties -- just not the version we're using in 1.2.x With Quarkus upgrade you should be able to use that ootb. Till then, JAVA_OPTIONS env var is your best option:
- https://github.com/Apicurio/apicurio-registry#security
The next release (1.3.0.Final) will include a newer version of Quarkus. I'm hoping to have that release done this week. I've merged the changes from the 1.2.x
branch and am hoping to also get Carles' new client merged as well prior to the release.
@EricWittmann @alesj Thanks for the information. I thought about trying to make apicurio take those non trivial java system properties from the dedicated env variables instead of using a single JAVA_OPTIONS env variable. If 1.3 will provide a file based configuration, that will be much better and flexible. That sounds good.
I did manage to get this to work by mapping in an config/application.properties from a Configmap into the Docker container's filesystem using Kubernetes, so I think the version of quarkus in use actually do support this.
@forsberg How did you managed to use the config/application.properties
file? Where should be located in the container?
I'm also trying to do that but with no luck, see #1805
@PlugaruT - this was a long time ago, and I'm not actively using Apicurio anymore, but digging into my chart it looks like I have a configmap with the application.properties
as key, and I mount it under /deployments/config
, but that's probably as much information I have for you.
Reading Quarkus Configuration Reference this seems valid if we assume that /deployments
is the working directory where the registry is started (see point 4 in the list of config sources), and if I shell into the pod I actually had running, this is the case.
So, in essence, mount your application.properties
so that it becomes $PWD/conf/application.properties
and Quarkus will read it.
@forsberg Thanks a lot, it seems to work if I mount it under /deployments
. Right now, if I exec into the container, the $PWD
is pointing to /home/jboss
and I was mounting it there...
@PlugaruT - glad you made it work! :+1:
As a general observation, sometimes having a look at /proc/<pid>
to see where /proc/<pid>/pwd
points to can be enlightening when it comes to the real working directory of a program.
Note to @jsenko and @carlesarnal
Took me a while to figure out, but setting these environment variables get you going (sample for AWS MSK)
REGISTRY_KAFKA_COMMON_SECURITY_PROTOCOL: SASL_SSL
REGISTRY_KAFKA_COMMON_SASL_JAAS_CONFIG: "software.amazon.msk.auth.iam.IAMLoginModule required;"
REGISTRY_KAFKA_COMMON_SASL_MECHANISM: AWS_MSK_IAM
REGISTRY_KAFKA_COMMON_SASL_CLIENT_CALLBACK_HANDLER_CLASS: software.amazon.msk.auth.iam.IAMClientCallbackHandler
We've discussed this issue and decided on supporting management of environment variables in the Apicurio CR like so:
spec:
configuration:
persistence: kafkasql
kafkasql:
bootstrapServers: "my-cluster-kafka-bootstrap.registry-example-kafkasql-scram.svc:9093"
env:
- key: REGISTRY_KAFKA_COMMON_SECURITY_PROTOCOL
value: SASL_SSL
- key: REGISTRY_KAFKA_COMMON_SASL_JAAS_CONFIG
value: "software.amazon.msk.auth.iam.IAMLoginModule required;"
- key: REGISTRY_KAFKA_COMMON_SASL_MECHANISM
value: AWS_MSK_IAM
- key: REGISTRY_KAFKA_COMMON_SASL_CLIENT_CALLBACK_HANDLER_CLASS
value: software.amazon.msk.auth.iam.IAMClientCallbackHandler
Solving https://github.com/Apicurio/apicurio-registry-operator/issues/167 and adding documentation about available env. variables.