Kafka Lag Exporter container fails with PKCS12 certificates when generated with JDK17

[jeqo]

Describe the bug PKCS12 keystores are the default since Java 9[1]. Even though JDK 8 supports PKCS12, I found that when creating keystores with newer versions (e.g. 17) the container running JDK 8 is not able to read the keystore:

2021-11-04 12:02:19,721 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-dev-cluster - Supervisor RestartSupervisor saw failure: Failed to construct kafka consumer org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:825)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
        at com.lightbend.kafkalagexporter.KafkaClient$.createConsumerClient(KafkaClient.scala:82)
        at com.lightbend.kafkalagexporter.KafkaClient$.apply(KafkaClient.scala:32)
        at com.lightbend.kafkalagexporter.MainApp$.$anonfun$start$1(MainApp.scala:32)
        at com.lightbend.kafkalagexporter.ConsumerGroupCollector$.$anonfun$init$1(ConsumerGroupCollector.scala:117)
        at akka.actor.typed.internal.BehaviorImpl$DeferredBehavior$$anon$1.apply(BehaviorImpl.scala:119)
        at akka.actor.typed.Behavior$.start(Behavior.scala:168)
        at akka.actor.typed.internal.RestartSupervisor.restartCompleted(Supervision.scala:372)
        at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:236)
        at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85)
        at akka.actor.typed.Behavior$.interpret(Behavior.scala:274)
        at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230)
        at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:130)
        at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:106)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:74)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:743)
        ... 24 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
        ... 28 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144)
        ... 31 common frames omitted
Caused by: java.io.IOException: parseAlgParameters failed: ObjectIdentifier() -- data isn't an object ID (tag = 48)
        at sun.security.pkcs12.PKCS12KeyStore.parseAlgParameters(PKCS12KeyStore.java:819)
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2027)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289)
        ... 32 common frames omitted
Caused by: java.io.IOException: ObjectIdentifier() -- data isn't an object ID (tag = 48)
        at sun.security.util.ObjectIdentifier.<init>(ObjectIdentifier.java:285)
        at sun.security.util.DerInputStream.getOID(DerInputStream.java:320)
        at com.sun.crypto.provider.PBES2Parameters.engineInit(PBES2Parameters.java:267)
        at java.security.AlgorithmParameters.init(AlgorithmParameters.java:293)
        at sun.security.pkcs12.PKCS12KeyStore.parseAlgParameters(PKCS12KeyStore.java:815)
        ... 35 common frames omitted

[1] https://openjdk.java.net/jeps/229

To Reproduce

• Create a PKCS12 keystore with JDK17.
• Mount keystore and configuration to Kafka Lag Exporter container.

Java versions tested: Container Java version:

openjdk version "1.8.0_265"
OpenJDK Runtime Environment (build 1.8.0_265-b01)
OpenJDK 64-Bit Server VM (build 25.265-b01, mixed mode)

Laptop Java version:

openjdk version "17" 2021-09-14
OpenJDK Runtime Environment (build 17+35-2724)
OpenJDK 64-Bit Server VM (build 17+35-2724, mixed mode, sharing)

Configuration:

kafka-lag-exporter {
  port = 9999

  client-group-id = "kafkaLagExporter"
  lookup-table-size = 120

  clusters = [
    {
      name = "dev-cluster"
      bootstrap-brokers = "kafka1:11091,kafka2:11092"

      admin-client-properties = {
        client.id = "admin-client-id"
        security.protocol = "SSL"
        ssl.truststore.location = "/etc/kafka/secrets/kafka.kafkaLagExporter.truststore.jks"
        ssl.truststore.password = "confluent"
        ssl.keystore.location = "/etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks"
        ssl.keystore.password = "confluent"
        ssl.keystore.type = "PKCS12"
        ssl.key.password = "confluent"
      }

      consumer-properties = {
        client.id = "consumer-client-id"
        security.protocol = "SSL"
        ssl.truststore.location = "/etc/kafka/secrets/kafka.kafkaLagExporter.truststore.jks"
        ssl.truststore.password = "confluent"
        ssl.keystore.location = "/etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks"
        ssl.keystore.password = "confluent"
        ssl.keystore.type = "PKCS12"
        ssl.key.password = "confluent"
      }
    }
  ]
}

Environment

• Version: 0.6.7
• Version of Apache Kafka cluster: 2.8.1
• Run with Helm or [x]Standalone

Additional context Add any other context about the problem here.

[jeqo]

Just tested with JDK11 generated keystores and fail on the same way:

2021-11-05 12:50:24,228 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-dev-cluster - Supervisor RestartSupervisor saw failure: Failed to construct kafka consumer org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:825)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
        at com.lightbend.kafkalagexporter.KafkaClient$.createConsumerClient(KafkaClient.scala:82)
        at com.lightbend.kafkalagexporter.KafkaClient$.apply(KafkaClient.scala:32)
        at com.lightbend.kafkalagexporter.MainApp$.$anonfun$start$1(MainApp.scala:32)
        at com.lightbend.kafkalagexporter.ConsumerGroupCollector$.$anonfun$init$1(ConsumerGroupCollector.scala:117)
        at akka.actor.typed.internal.BehaviorImpl$DeferredBehavior$$anon$1.apply(BehaviorImpl.scala:119)
        at akka.actor.typed.Behavior$.start(Behavior.scala:168)
        at akka.actor.typed.internal.RestartSupervisor.restartCompleted(Supervision.scala:372)
        at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:236)
        at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85)
        at akka.actor.typed.Behavior$.interpret(Behavior.scala:274)
        at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230)
        at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:130)
        at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:106)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:577)
        at akka.actor.ActorCell.invoke(ActorCell.scala:547)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type JKS
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:74)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:743)
        ... 24 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type JKS
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
        ... 28 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type JKS
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144)
        ... 31 common frames omitted
Caused by: java.io.IOException: Invalid keystore format
        at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:666)
        at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:57)
        at sun.security.provider.KeyStoreDelegator.engineLoad(KeyStoreDelegator.java:224)
        at sun.security.provider.JavaKeyStore$DualFormatJKS.engineLoad(JavaKeyStore.java:71)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289)
        ... 32 common frames omitted

gitpod /workspace/jmx-monitoring-stacks $ java -version
Picked up JAVA_TOOL_OPTIONS: -Xmx2576m
openjdk version "11.0.13" 2021-10-19 LTS
OpenJDK Runtime Environment Zulu11.52+13-CA (build 11.0.13+8-LTS)
OpenJDK 64-Bit Server VM Zulu11.52+13-CA (build 11.0.13+8-LTS, mixed mode)

[timtebeek]

Not sure if this 'll work in your case, but at work we had to add a RUN yum update -y to our Dockerfile.

Which is weird since that's also already in the Dockerfile generated. https://github.com/lightbend/kafka-lag-exporter/blob/master/build.sbt#L55

Maybe a new tag would already be enough to fix the issue?

Although ideally swap Java 8 for 11, also given the support timeline for 8.

[jeqo]

Thanks for the workaround @timtebeek !

Maybe a new tag would already be enough to fix the issue? Although ideally swap Java 8 for 11, also given the support timeline for 8.

Agree. I was thinking the best solution could be around those lines.

cc @seglo

[slachiewicz]

Yes, update to latest JDK will fix issue - root cause is https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8267599

[seglo]

I just released 0.6.8. LMK if that addresses it.

[jeqo]

@seglo thanks for the heads up! Unfortunately, it's not fixing the issue:

2021-11-22 12:02:04,671 ERROR c.l.k.ConsumerGroupCollector$ akka://kafka-lag-exporter/user/consumer-group-collector-dev-cluster - Supervisor RestartSupervisor saw failure [3]: Failed to construct kafka consumer org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:825)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:666)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:646)
        at com.lightbend.kafkalagexporter.KafkaClient$.createConsumerClient(KafkaClient.scala:82)
        at com.lightbend.kafkalagexporter.KafkaClient$.apply(KafkaClient.scala:32)
        at com.lightbend.kafkalagexporter.MainApp$.$anonfun$start$1(MainApp.scala:32)
        at com.lightbend.kafkalagexporter.ConsumerGroupCollector$.$anonfun$init$1(ConsumerGroupCollector.scala:117)
        at akka.actor.typed.internal.BehaviorImpl$DeferredBehavior$$anon$1.apply(BehaviorImpl.scala:120)
        at akka.actor.typed.Behavior$.start(Behavior.scala:168)
        at akka.actor.typed.internal.RestartSupervisor.restartCompleted(Supervision.scala:383)
        at akka.actor.typed.internal.RestartSupervisor.aroundReceive(Supervision.scala:243)
        at akka.actor.typed.internal.InterceptorImpl.receive(InterceptorImpl.scala:85)
        at akka.actor.typed.Behavior$.interpret(Behavior.scala:274)
        at akka.actor.typed.Behavior$.interpretMessage(Behavior.scala:230)
        at akka.actor.typed.internal.adapter.ActorAdapter.handleMessage(ActorAdapter.scala:131)
        at akka.actor.typed.internal.adapter.ActorAdapter.aroundReceive(ActorAdapter.scala:107)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:580)
        at akka.actor.ActorCell.invoke(ActorCell.scala:548)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:270)
        at akka.dispatch.Mailbox.run(Mailbox.scala:231)
        at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
        at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
        at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
        at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
        at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:74)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:157)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:73)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:105)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:743)
        ... 24 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:163)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.<init>(SslEngineBuilder.java:104)
        at org.apache.kafka.common.security.ssl.SslFactory.configure(SslFactory.java:95)
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:72)
        ... 28 common frames omitted
Caused by: org.apache.kafka.common.KafkaException: Failed to load SSL keystore /etc/kafka/secrets/kafka.kafkaLagExporter.keystore.jks of type PKCS12
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:292)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder.createSSLContext(SslEngineBuilder.java:144)
        ... 31 common frames omitted
Caused by: java.io.IOException: Integrity check failed: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2129)
        at java.security.KeyStore.load(KeyStore.java:1445)
        at org.apache.kafka.common.security.ssl.SslEngineBuilder$SecurityStore.load(SslEngineBuilder.java:289)
        ... 32 common frames omitted
Caused by: java.security.NoSuchAlgorithmException: Algorithm HmacPBESHA256 not available
        at javax.crypto.Mac.getInstance(Mac.java:181)
        at sun.security.pkcs12.PKCS12KeyStore.engineLoad(PKCS12KeyStore.java:2107)
        ... 34 common frames omitted

(same as https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8267599)

docker ps
da95b1fe3a8f   lightbend/kafka-lag-exporter:0.6.8                         "/opt/docker/bin/kaf…"   16 minutes ago   Up 16 minutes                                                                                                                                            kafka-lag-exporter

docker exec kafka-lag-exporter java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)

[seglo]

Ok. I created a ticket for Java 11. Maybe that will help. https://github.com/lightbend/kafka-lag-exporter/issues/286

[timtebeek]

Not sure if this 'll work in your case, but at work we had to add a RUN yum update -y to our Dockerfile.

As of today this workaround no longer works, as CentOS Linux 8 is End Of Life. There's a further vault workaround posted here, but ideally this is fixed at the root.

[jeqo]

Haven't tested it, but seems like #297 already solves this?

[seglo]

I'll be cutting a release shortly once I sort out the release process. Thanks for your patience.

[seglo]

Kafka Lag Exporter v0.7.0 has been released. Please let me know how it goes!

[sverrehu]

Is this still an issue, now that the container uses Java 17? I suggest closing it.