Unable to add custom jars to KSQL_CLASSPATH

[vcrfxia]

Describe the bug

A user is unable to load a custom jar onto KSQL_CLASSPATH via building a new Docker image with the custom jars added to the /usr/share/java/ksqldb-rest-app/ which should be on the classpath. See https://github.com/confluentinc/ksql/issues/6772 for Dockerfile and details.

To Reproduce

See https://github.com/confluentinc/ksql/issues/6772.

Expected behavior

Custom jar should be findable by ksqlDB server.

Actual behaviour

An error message indicating that the custom jar is not found.

[vcrfxia]

Hey @shane-blacklane, can you verify that the permissions on the custom jars you've loaded into your docker image are accessible by the user the ksqlDB server is running as? In building the base docker image for the ksqlDB server, we ensure this is the case: https://github.com/confluentinc/ksql/blob/v0.11.0-ksqldb/ksqldb-docker/Dockerfile#L13 I wonder if something similar needs to be done in your dockerfile as well.

Another good thing to sanity check is the KSQL_CLASSPATH in the command that launches the ksqlDB server within your docker image. From your dockerfile, it appears the directory you've added the jars to should already be contained in the default KSQL_CLASSPATH but it would be good to confirm.

[shane-blacklane]

Hi @vcrfxia thanks for the suggestions :+1:

I checked the ownership of the jars, and you were right they weren't owned by the appuser so i added --chown to the COPY my dockerfile to change that, however the issue still persists.

For the KSQL_CLASSPATH, the dockerfile you linked doesn't define that variable, however this one for example does. I also can't find the actual dockerfile for the actual image I'm using. When i run printenv in my image I don't get KSQL_CLASSPATH in the output.

EDIT: I also realised that I maybe wasn't copying the jars in to the right location (was copying them to /usr/share/java/ksqldb-rest-app/) but in this they go to /usr/share/java/ksqldb-server/* so I tried that too but no luck :cry:

[vcrfxia]

Did you check what user the ksqlDB server is running as?

KSQL_CLASSPATH is not defined as an env var. Rather, it's part of the command that launches the ksqlDB server. If you list Java processes and identify the command for the ksqlDB server, you should be able to see it's value and verify the jars are in a directory on the classpath.

[Ocramius]

FWIW, I just tried this myself:

FROM confluentinc/cp-ksqldb-server:7.8.0

# https://github.com/confluentinc/ksql/issues/6831#issuecomment-758770592
# https://github.com/confluentinc/ksql/pull/10258
ADD --chown=appuser https://packages.confluent.io/maven/io/confluent/ksql/ksqldb-rocksdb-config-setter/7.8.0/ksqldb-rocksdb-config-setter-7.8.0.jar /usr/share/java/ksqldb-server/

docker build -t ksql-dummy .
docker run --rm -ti ksql-dummy bash
echo $KSQL_CLASSPATH | grep ksqldb-rocksdb-config-setter-7.8.0.jar # works?

[nicollette]

I ran into a similar issue and thought it'd be useful to share the workaround I found.

I have a Docker image that extends confluentinc/cp-ksqldb-server:7.1.16-1-ubi8.amd64 and includes adding a custom JAR containing the class needed for sasl.login.callback.handler.class. I ensured that my Dockerfile copied the JAR to a directory in the classpath and the ksqldb user had access to the JAR. The ksqldb-server would fail at startup with:

===> User
uid=2000(appuser) gid=2000(appuser) groups=2000(appuser),2800
===> Configuring ...
KSQL_OPTS should contain the 'com.sun.management.jmxremote.rmi.port' property. It is required for accessing the JMX metrics externally. Also remember to map the JMX port so that it is accessible from outside the container.
===> Preparing KSQL Server properties
===> Prepared KSQL Server properties - /etc/ksqldb-server/ksqldb-server.properties, /etc/ksqldb-server/connect.properties, /etc/ksqldb-server/log4j.properties
===> Running preflight checks ...
===> Check if Kafka is healthy ...
[main] ERROR io.confluent.admin.utils.cli.KafkaReadyCommand - Error while running kafka-ready.
org.apache.kafka.common.config.ConfigException: Invalid value com.etsy.gcp.oauth.client.OAuthBearerTokenCallbackHandler for configuration sasl.login.callback.handler.class: Class gcp.oauth.client.OAuthBearerTokenCallbackHandler could not be found.
        at org.apache.kafka.common.config.ConfigDef.parseType(ConfigDef.java:744)
        at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:490)
        at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:483)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:115)
        at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:148)
        at org.apache.kafka.clients.admin.AdminClientConfig.<init>(AdminClientConfig.java:231)
        at org.apache.kafka.clients.admin.Admin.create(Admin.java:143)
        at org.apache.kafka.clients.admin.AdminClient.create(AdminClient.java:49)
        at io.confluent.admin.utils.ClusterStatus.isKafkaReady(ClusterStatus.java:136)
        at io.confluent.admin.utils.cli.KafkaReadyCommand.main(KafkaReadyCommand.java:149)

I discovered that the io.confluent.admin.utils.cli.KafkaReadyCommand is executed in the cp-ksqldb-server/include/etc/confluent/docker/ensure script. Adding my custom JAR to /usr/share/java/cp-base-new/ fixed the error.

This is the Dockerfile that worked for me:

FROM confluentinc/cp-ksqldb-server:7.1.16-1-ubi8.amd64

USER root

COPY --chown=appuser:appuser kafka-oauth/kafka-oauth-client/target/kafka-oauth-client-*-shaded-jar.jar /usr/share/java/ksqldb-rest-app/
ENV KSQL_CLASSPATH=/usr/share/java/ksqldb-rest-app/*

# This is required so that the /etc/confluent/docker/ensure startup script can access the oauth JAR in authenticated clients: https://github.com/confluentinc/ksql-images/blob/7a81e14934900f8e022ed1275a2e1f5446866803/cp-ksqldb-server/include/etc/confluent/docker/ensure#L31
COPY --chown=appuser:appuser kafka-oauth/kafka-oauth-client/target/kafka-oauth-client-*-shaded-jar.jar /usr/share/java/cp-base-new/

USER appuser