schema-registry
schema-registry copied to clipboard
Published
20 hours ago •
confluentinc
confluentinc
Schema-Registry doesn't connect to SSL kafka cluster
I'm trying to deploy the latest schema-registry docker image on k8s and trying to connect it on a SSL secured kafka cluster. I'm mounting the CA and user cert and defining parameters as ENVs:
- env:
- name: SCHEMA_REGISTRY_KAFKASTORE_SECURITY_PROTOCOL
value: SSL
- name: SCHEMA_REGISTRY_KAFKASTORE_SSL_TRUSTSTORE_LOCATION
value: /var/certs/user-truststore.jks
- name: SCHEMA_REGISTRY_KAFKASTORE_SSL_TRUSTSTORE_PASSWORD
value: <hidden>
- name: SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_LOCATION
value: /var/certs/user-keystore.p12
- name: SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_PASSWORD
value: <hidden>
- name: SCHEMA_REGISTRY_KAFKASTORE_SSL_KEY_PASSWORD
value: <hidden>
- name: SCHEMA_REGISTRY_LISTENERS
value: http://0.0.0.0:8081
- name: SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS
value: SSL://kafka-bootstrap:9093
- name: SCHEMA_REGISTRY_KAFKASTORE_GROUP_ID
value: schema-registry
- name: SCHEMA_REGISTRY_MASTER_ELIGIBILITY
value: "true"
- name: SCHEMA_REGISTRY_KAFKASTORE_SSL_KEYSTORE_TYPE
value: PKCS12
- name: SCHEMA_REGISTRY_HOST_NAME
value: cp-schema-registry-server
- name: SCHEMA_REGISTRY_HEAP_OPTS
value: -Xms512M -Xmx512M
- name: JMX_PORT
value: "5555"
As seen here from the container logs it is not complaining about them:
===> User
uid=1000(appuser) gid=1000(appuser) groups=1000(appuser)
===> Configuring ...
===> Running preflight checks ...
===> Check if Kafka is healthy ...
[main] INFO org.apache.kafka.clients.admin.AdminClientConfig - AdminClientConfig values:
bootstrap.servers = [SSL://kafka-bootstrap:9093]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = [hidden]
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = /var/certs/user-keystore.p12
ssl.keystore.password = [hidden]
ssl.keystore.type = PKCS12
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = /var/certs/user-truststore.jks
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'ssl.keystore.type' was supplied but isn't a known config.
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'ssl.truststore.location' was supplied but isn't a known config.
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'ssl.keystore.password' was supplied but isn't a known config.
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'ssl.key.password' was supplied but isn't a known config.
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'group.id' was supplied but isn't a known config.
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'ssl.keystore.location' was supplied but isn't a known config.
[main] WARN org.apache.kafka.clients.admin.AdminClientConfig - The configuration 'ssl.truststore.password' was supplied but isn't a known config.
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka version: 6.0.2-ccs
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka commitId: a58736d0602d24aa
[main] INFO org.apache.kafka.common.utils.AppInfoParser - Kafka startTimeMs: 1615214096609
===> Launching ...
===> Launching schema-registry ...
[2021-03-08 14:35:34,106] INFO SchemaRegistryConfig values:
access.control.allow.headers =
access.control.allow.methods =
access.control.allow.origin =
access.control.skip.options = true
authentication.method = NONE
authentication.realm =
authentication.roles = [*]
authentication.skip.paths = []
avro.compatibility.level =
compression.enable = true
csrf.prevention.enable = false
csrf.prevention.token.endpoint = /csrf
csrf.prevention.token.expiration.minutes = 30
csrf.prevention.token.max.entries = 10000
debug = false
host.name = cp-schema-registry-server
idle.timeout.ms = 30000
inter.instance.headers.whitelist = []
inter.instance.protocol = http
kafkastore.bootstrap.servers = [SSL://kafka-bootstrap:9093]
kafkastore.connection.url =
kafkastore.group.id = schema-registry
kafkastore.init.timeout.ms = 60000
kafkastore.sasl.kerberos.kinit.cmd = /usr/bin/kinit
kafkastore.sasl.kerberos.min.time.before.relogin = 60000
kafkastore.sasl.kerberos.service.name =
kafkastore.sasl.kerberos.ticket.renew.jitter = 0.05
kafkastore.sasl.kerberos.ticket.renew.window.factor = 0.8
kafkastore.sasl.mechanism = GSSAPI
kafkastore.security.protocol = SSL
kafkastore.ssl.cipher.suites =
kafkastore.ssl.enabled.protocols = TLSv1.2,TLSv1.1,TLSv1
kafkastore.ssl.endpoint.identification.algorithm =
kafkastore.ssl.key.password = [hidden]
kafkastore.ssl.keymanager.algorithm = SunX509
kafkastore.ssl.keystore.location = /var/certs/user-keystore.p12
kafkastore.ssl.keystore.password = [hidden]
kafkastore.ssl.keystore.type = PKCS12
kafkastore.ssl.protocol = TLS
kafkastore.ssl.provider =
kafkastore.ssl.trustmanager.algorithm = PKIX
kafkastore.ssl.truststore.location = /var/certs/user-truststore.jks
kafkastore.ssl.truststore.password = [hidden]
kafkastore.ssl.truststore.type = JKS
kafkastore.timeout.ms = 500
kafkastore.topic = _schemas
kafkastore.topic.replication.factor = 3
kafkastore.update.handlers = []
kafkastore.write.max.retries = 5
kafkastore.zk.session.timeout.ms = 30000
leader.eligibility = true
listeners = [http://0.0.0.0:8081]
master.eligibility = true
metric.reporters = []
metrics.jmx.prefix = kafka.schema.registry
metrics.num.samples = 2
metrics.sample.window.ms = 30000
metrics.tag.map = []
mode.mutability = false
port = 8081
request.logger.name = io.confluent.rest-utils.requests
request.queue.capacity = 2147483647
request.queue.capacity.growby = 64
request.queue.capacity.init = 128
resource.extension.class = []
resource.extension.classes = []
resource.static.locations = []
response.http.headers.config =
response.mediatype.default = application/vnd.schemaregistry.v1+json
response.mediatype.preferred = [application/vnd.schemaregistry.v1+json, application/vnd.schemaregistry+json, application/json]
rest.servlet.initializor.classes = []
schema.compatibility.level = backward
schema.providers = []
schema.registry.group.id = schema-registry
schema.registry.inter.instance.protocol =
schema.registry.resource.extension.class = []
schema.registry.zk.namespace = schema_registry
shutdown.graceful.ms = 1000
ssl.cipher.suites = []
ssl.client.auth = false
ssl.client.authentication = NONE
ssl.enabled.protocols = []
ssl.endpoint.identification.algorithm = null
ssl.key.password = [hidden]
ssl.keymanager.algorithm =
ssl.keystore.location =
ssl.keystore.password = [hidden]
ssl.keystore.reload = false
ssl.keystore.type = JKS
ssl.keystore.watch.location =
ssl.protocol = TLS
ssl.provider =
ssl.trustmanager.algorithm =
ssl.truststore.location =
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
thread.pool.max = 200
thread.pool.min = 8
websocket.path.prefix = /ws
websocket.servlet.initializor.classes = []
zookeeper.set.acl = false
(io.confluent.kafka.schemaregistry.rest.SchemaRegistryConfig)
[2021-03-08 14:35:35,709] INFO Logging initialized @19601ms to org.eclipse.jetty.util.log.Slf4jLog (org.eclipse.jetty.util.log)
[2021-03-08 14:35:36,106] INFO Initial capacity 128, increased by 64, maximum capacity 2147483647. (io.confluent.rest.ApplicationServer)
[2021-03-08 14:35:39,009] INFO Adding listener: http://0.0.0.0:8081 (io.confluent.rest.ApplicationServer)
[2021-03-08 14:35:46,908] INFO AdminClientConfig values:
bootstrap.servers = [SSL://kafka-bootstrap:9093]
client.dns.lookup = use_all_dns_ips
client.id =
connections.max.idle.ms = 300000
default.api.timeout.ms = 60000
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
receive.buffer.bytes = 65536
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = SSL
security.providers = null
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
ssl.endpoint.identification.algorithm = https
ssl.engine.factory.class = null
ssl.key.password = [hidden]
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = /var/certs/user-keystore.p12
ssl.keystore.password = [hidden]
ssl.keystore.type = PKCS12
ssl.protocol = TLSv1.3
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = /var/certs/user-truststore.jks
ssl.truststore.password = [hidden]
ssl.truststore.type = JKS
Still getting this error:
[2021-03-08 14:36:17,306] ERROR Error starting the schema registry (io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication)
io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryException: Failed to get Kafka cluster ID
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.kafkaClusterId(KafkaSchemaRegistry.java:1228)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.<init>(KafkaSchemaRegistry.java:156)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.initSchemaRegistry(SchemaRegistryRestApplication.java:69)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryRestApplication.configureBaseApplication(SchemaRegistryRestApplication.java:88)
at io.confluent.rest.Application.configureHandler(Application.java:255)
at io.confluent.rest.ApplicationServer.doStart(ApplicationServer.java:196)
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
at io.confluent.kafka.schemaregistry.rest.SchemaRegistryMain.main(SchemaRegistryMain.java:43)
Caused by: java.util.concurrent.TimeoutException
at org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:108)
at org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:272)
at io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.kafkaClusterId(KafkaSchemaRegistry.java:1226)
... 7 more
Interesting... did you get any updates on that ? I have the same problem on a Kafka Strimzi setup with Authorization enabled.
I'm also having this problem but with PLAINTEXT. Increasing resources seems to work at least for now. It's an intermittent issue for me though
@cortopy thanks to your comment we removed resources limits on our deployment and it started to work!
I'm also having this problem but with PLAINTEXT. Increasing resources seems to work at least for now. It's an intermittent issue for me though
How did you increase the number of resources. I am getting the error which says failed to get the kafka cluster id
The keystores listed here as blank -- this does not turn out to be important. The settings from the configuration files are used anyway.
In my case they did not because I failed to restart the correct service... the broker systemd service is called confluent-kafka not confluent-kafka-broker, which brakes the pattern from all other kafka services.
