Topics not loading for OCI Streaming

[linkedby-data]

Describe the bug Hi, I'm trying to use kafka-ui v0.3.3 connected with OCI Streaming, but the topics and consumers don't load, only the broker dashboard displays information.

Set up Run latest image in docker:

version: '2'

services:
  kafka-ui:
    container_name: kafka-ui
    image: provectuslabs/kafka-ui:latest
    ports:
      - 8080:8080
    environment:
      KAFKA_CLUSTERS_0_NAME: kafka-oci
      KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION: 'true'
      KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS: server:9092
      KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL: SASL_SSL
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM: PLAIN
      KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG: 'org.apache.kafka.common.security.plain.PlainLoginModule required username="username" password="password;'

Steps to Reproduce Steps to reproduce the behavior:

1. Run kafka-ui (docker-compose up / http://localhost:8080).
2. Select Topics from the menu and wait for a while. Topics are not listed and the error is shown in the log.

Expected behavior Cluster topic list.

Screenshots

Additional context Log with erros:

Starting kafka-ui ... done
Attaching to kafka-ui
kafka-ui    | 
kafka-ui    |   _  __      __ _               _   _ ___
kafka-ui    |  | |/ /__ _ / _| | ____ _      | | | |_ _|
kafka-ui    |  | ' // _` | |_| |/ / _` |_____| | | || |
kafka-ui    |  | . \ (_| |  _|   < (_| |_____| |_| || |
kafka-ui    |  |_|\_\__,_|_| |_|\_\__,_|      \___/|___|
kafka-ui    | 
kafka-ui    | 
kafka-ui    | 2022-03-05 01:06:42,111 INFO  [background-preinit] o.h.v.i.u.Version: HV000001: Hibernate Validator 6.2.0.Final
kafka-ui    | 2022-03-05 01:06:42,144 INFO  [main] c.p.k.u.KafkaUiApplication: Starting KafkaUiApplication using Java 13.0.9 on 9d542612eb28 with PID 1 (/kafka-ui-api.jar started by root in /)
kafka-ui    | 2022-03-05 01:06:42,147 DEBUG [main] c.p.k.u.KafkaUiApplication: Running with Spring Boot v2.5.6, Spring v5.3.12
kafka-ui    | 2022-03-05 01:06:42,147 INFO  [main] c.p.k.u.KafkaUiApplication: No active profile set, falling back to default profiles: default
kafka-ui    | 2022-03-05 01:06:43,333 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Bootstrapping Spring Data LDAP repositories in DEFAULT mode.
kafka-ui    | 2022-03-05 01:06:43,376 INFO  [main] o.s.d.r.c.RepositoryConfigurationDelegate: Finished Spring Data repository scanning in 36 ms. Found 0 LDAP repository interfaces.
kafka-ui    | 2022-03-05 01:06:44,069 INFO  [main] o.s.b.a.s.r.ReactiveUserDetailsServiceAutoConfiguration: 
kafka-ui    | 
kafka-ui    | Using generated security password: 05a5da84-5d29-4dd6-891a-cdb0d00449e5
kafka-ui    | 
kafka-ui    | 2022-03-05 01:06:44,260 INFO  [main] c.p.k.u.s.DeserializationService: Using SchemaRegistryAwareRecordSerDe for cluster 'kafka-oci'
kafka-ui    | 2022-03-05 01:06:44,769 INFO  [main] o.s.b.a.e.w.EndpointLinksResolver: Exposing 2 endpoint(s) beneath base path '/actuator'
kafka-ui    | 2022-03-05 01:06:44,884 WARN  [main] c.p.k.u.c.a.DisabledAuthSecurityConfig: Authentication is disabled. Access will be unrestricted.
kafka-ui    | 2022-03-05 01:06:45,041 INFO  [main] o.s.l.c.s.AbstractContextSource: Property 'userDn' not set - anonymous context will be used for read-write operations
kafka-ui    | 2022-03-05 01:06:45,280 INFO  [main] o.s.b.w.e.n.NettyWebServer: Netty started on port 8080
kafka-ui    | 2022-03-05 01:06:45,301 INFO  [main] c.p.k.u.KafkaUiApplication: Started KafkaUiApplication in 3.906 seconds (JVM running for 4.48)
kafka-ui    | 2022-03-05 01:06:45,323 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: kafka-oci
kafka-ui    | 2022-03-05 01:06:45,337 INFO  [parallel-1] o.a.k.c.a.AdminClientConfig: AdminClientConfig values: 
kafka-ui    | 	bootstrap.servers = [server:9092]
kafka-ui    | 	client.dns.lookup = use_all_dns_ips
kafka-ui    | 	client.id = 
kafka-ui    | 	connections.max.idle.ms = 300000
kafka-ui    | 	default.api.timeout.ms = 60000
kafka-ui    | 	metadata.max.age.ms = 300000
kafka-ui    | 	metric.reporters = []
kafka-ui    | 	metrics.num.samples = 2
kafka-ui    | 	metrics.recording.level = INFO
kafka-ui    | 	metrics.sample.window.ms = 30000
kafka-ui    | 	receive.buffer.bytes = 65536
kafka-ui    | 	reconnect.backoff.max.ms = 1000
kafka-ui    | 	reconnect.backoff.ms = 50
kafka-ui    | 	request.timeout.ms = 30000
kafka-ui    | 	retries = 2147483647
kafka-ui    | 	retry.backoff.ms = 100
kafka-ui    | 	sasl.client.callback.handler.class = null
kafka-ui    | 	sasl.jaas.config = [hidden]
kafka-ui    | 	sasl.kerberos.kinit.cmd = /usr/bin/kinit
kafka-ui    | 	sasl.kerberos.min.time.before.relogin = 60000
kafka-ui    | 	sasl.kerberos.service.name = null
kafka-ui    | 	sasl.kerberos.ticket.renew.jitter = 0.05
kafka-ui    | 	sasl.kerberos.ticket.renew.window.factor = 0.8
kafka-ui    | 	sasl.login.callback.handler.class = null
kafka-ui    | 	sasl.login.class = null
kafka-ui    | 	sasl.login.refresh.buffer.seconds = 300
kafka-ui    | 	sasl.login.refresh.min.period.seconds = 60
kafka-ui    | 	sasl.login.refresh.window.factor = 0.8
kafka-ui    | 	sasl.login.refresh.window.jitter = 0.05
kafka-ui    | 	sasl.mechanism = PLAIN
kafka-ui    | 	security.protocol = SASL_SSL
kafka-ui    | 	security.providers = null
kafka-ui    | 	send.buffer.bytes = 131072
kafka-ui    | 	socket.connection.setup.timeout.max.ms = 30000
kafka-ui    | 	socket.connection.setup.timeout.ms = 10000
kafka-ui    | 	ssl.cipher.suites = null
kafka-ui    | 	ssl.enabled.protocols = [TLSv1.2, TLSv1.3]
kafka-ui    | 	ssl.endpoint.identification.algorithm = https
kafka-ui    | 	ssl.engine.factory.class = null
kafka-ui    | 	ssl.key.password = null
kafka-ui    | 	ssl.keymanager.algorithm = SunX509
kafka-ui    | 	ssl.keystore.certificate.chain = null
kafka-ui    | 	ssl.keystore.key = null
kafka-ui    | 	ssl.keystore.location = null
kafka-ui    | 	ssl.keystore.password = null
kafka-ui    | 	ssl.keystore.type = JKS
kafka-ui    | 	ssl.protocol = TLSv1.3
kafka-ui    | 	ssl.provider = null
kafka-ui    | 	ssl.secure.random.implementation = null
kafka-ui    | 	ssl.trustmanager.algorithm = PKIX
kafka-ui    | 	ssl.truststore.certificates = null
kafka-ui    | 	ssl.truststore.location = null
kafka-ui    | 	ssl.truststore.password = null
kafka-ui    | 	ssl.truststore.type = JKS
kafka-ui    | 
kafka-ui    | 2022-03-05 01:06:45,441 INFO  [parallel-1] o.a.k.c.s.a.AbstractLogin: Successfully logged in.
kafka-ui    | 2022-03-05 01:06:45,581 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka version: 2.8.0
kafka-ui    | 2022-03-05 01:06:45,581 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka commitId: ebb1d6e21cc92130
kafka-ui    | 2022-03-05 01:06:45,581 INFO  [parallel-1] o.a.k.c.u.AppInfoParser: Kafka startTimeMs: 1646442405580
kafka-ui    | 2022-03-05 01:06:51,003 DEBUG [kafka-admin-client-thread | adminclient-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: kafka-oci
kafka-ui    | 2022-03-05 01:07:00,773 WARN  [parallel-1] o.h.v.i.p.j.JavaBeanExecutable: HV000254: Missing parameter metadata for TopicColumnsToSortDTO(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
kafka-ui    | 2022-03-05 01:07:15,297 DEBUG [parallel-4] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: kafka-oci
kafka-ui    | 2022-03-05 01:07:17,315 DEBUG [kafka-admin-client-thread | adminclient-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: kafka-oci
kafka-ui    | 2022-03-05 01:07:45,297 DEBUG [parallel-4] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: kafka-oci
kafka-ui    | 2022-03-05 01:07:48,138 DEBUG [kafka-admin-client-thread | adminclient-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: kafka-oci
kafka-ui    | 2022-03-05 01:08:01,487 WARN  [kafka-admin-client-thread | adminclient-1] c.p.k.u.e.ErrorCode: Multiple class com.provectus.kafka.ui.exception.ErrorCode values refer to code 4001
kafka-ui    | 2022-03-05 01:08:01,500 ERROR [kafka-admin-client-thread | adminclient-1] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler: [d45dacc0-6]  500 Server Error for HTTP GET "/api/clusters/kafka-oci/topics?showInternal=true&search=&orderBy=NAME"
kafka-ui    | org.apache.kafka.common.errors.TimeoutException: Call(callName=listOffsets on broker 11, deadlineMs=1646442481450, tries=1, nextAllowedTryMs=1646442481551) timed out at 1646442481451 after 1 attempt(s)
kafka-ui    | 	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
kafka-ui    | Error has been observed at the following site(s):
kafka-ui    | 	*__checkpoint ⇢ Handler com.provectus.kafka.ui.controller.TopicsController#getTopics(String, Integer, Integer, Boolean, String, TopicColumnsToSortDTO, ServerWebExchange) [DispatcherHandler]
kafka-ui    | 	*__checkpoint ⇢ com.provectus.kafka.ui.config.CustomWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ com.provectus.kafka.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
kafka-ui    | 	*__checkpoint ⇢ HTTP GET "/api/clusters/kafka-oci/topics?showInternal=true&search=&orderBy=NAME" [ExceptionHandlingWebHandler]
kafka-ui    | Stack trace:
kafka-ui    | Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listOffsets on broker 11
^Akafka-ui    | 2022-03-05 01:08:15,296 DEBUG [parallel-4] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: kafka-oci
kafka-ui    | 2022-03-05 01:08:16,503 DEBUG [kafka-admin-client-thread | adminclient-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: kafka-oci

[github-actions[bot]]

Hello there linkedby-data! 👋

Thank you and congratulations 🎉 for opening your very first issue in this project! 💖

In case you want to claim this issue, please comment down below! We will try to get back to you as soon as we can. 👀

[Haarolean]

Hey, thanks for reporting the bug.

We'll take a look.

[iliax]

Hello @linkedby-data ! Can you please share full compose file (excluding sensitive info)?

[github-actions[bot]]

This issue has been automatically marked as stale because no requested feedback has been provided. It will be closed if no further activity occurs. Thank you for your contributions.

[raafael]

Hello!

I have the same issue when list topics in OCI Streaming cluster. I am running docker provectuslabs/kafka-ui:latest @linkedby-data did you manage to solve it?

Config file:

kafka:
  clusters:
    - name: oracle-streaming-dev
      bootstrapServers: "hostname:9092"
      disableLogDirsCollection: true
      properties:
        sasl.mechanism: PLAIN
        security.protocol: SASL_SSL
        sasl.jaas.config: org.apache.kafka.common.security.plain.PlainLoginModule required username="user" password="pass";
        session.timeout.ms: 30000

Logs:

2022-07-19 12:25:32,114 WARN  [parallel-8] o.h.v.i.p.j.JavaBeanExecutable: HV000254: Missing parameter metadata for TopicColumnsToSortDTO(String, int, String), which declares implicit or synthetic parameters. Automatic resolution of generic type information for method parameters may yield incorrect results if multiple parameters have the same erasure. To solve this, compile your code with the '-parameters' flag.
2022-07-19 12:25:47,508 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: oracle-streaming-dev
2022-07-19 12:25:48,948 DEBUG [kafka-admin-client-thread | adminclient-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: oracle-streaming-dev
2022-07-19 12:26:17,507 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: oracle-streaming-dev
2022-07-19 12:26:18,915 DEBUG [kafka-admin-client-thread | adminclient-1] c.p.k.u.s.ClustersMetricsScheduler: Metrics updated for cluster: oracle-streaming-dev
2022-07-19 12:26:32,648 WARN  [kafka-admin-client-thread | adminclient-1] c.p.k.u.e.ErrorCode: Multiple class com.provectus.kafka.ui.exception.ErrorCode values refer to code 4001
2022-07-19 12:26:32,671 ERROR [kafka-admin-client-thread | adminclient-1] o.s.b.a.w.r.e.AbstractErrorWebExceptionHandler: [b52b882f-5]  500 Server Error for HTTP GET "/api/clusters/oracle-streaming-dev/topics?showInternal=true&search=&orderBy=NAME"
org.apache.kafka.common.errors.TimeoutException: Call(callName=listOffsets on broker 11, deadlineMs=1658233592610, tries=1, nextAllowedTryMs=1658233592711) timed out at 1658233592611 after 1 attempt(s)
        Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
        *__checkpoint ⇢ Handler com.provectus.kafka.ui.controller.TopicsController#getTopics(String, Integer, Integer, Boolean, String, TopicColumnsToSortDTO, ServerWebExchange) [DispatcherHandler]
        *__checkpoint ⇢ com.provectus.kafka.ui.config.CustomWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ com.provectus.kafka.ui.config.ReadOnlyModeFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.web.server.authorization.AuthorizationWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.web.server.authorization.ExceptionTranslationWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.web.server.authentication.logout.LogoutWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.web.server.savedrequest.ServerRequestCacheWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.web.server.context.SecurityContextServerWebExchangeWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.web.server.context.ReactorContextWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.web.server.header.HttpHeaderWriterWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.config.web.server.ServerHttpSecurity$ServerWebExchangeReactorContextWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.security.web.server.WebFilterChainProxy [DefaultWebFilterChain]
        *__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
        *__checkpoint ⇢ HTTP GET "/api/clusters/oracle-streaming-dev/topics?showInternal=true&search=&orderBy=NAME" [ExceptionHandlingWebHandler]
Stack trace:
Caused by: org.apache.kafka.common.errors.TimeoutException: Timed out waiting for a node assignment. Call: listOffsets on broker 11
2022-07-19 12:26:47,508 DEBUG [parallel-1] c.p.k.u.s.ClustersMetricsScheduler: Start getting metrics for kafkaCluster: oracle-streaming-dev

[iliax]

@linkedby-data, @raafael thank you for creating issue. I checked OCI streaming today:

docker run -p 8080:8080 \
        -e KAFKA_CLUSTERS_0_NAME=oci-example \
        -e KAFKA_CLUSTERS_0_BOOTSTRAPSERVERS=cell-1.streaming.eu-amsterdam-1.oci.oraclecloud.com:9092 \
        -e KAFKA_CLUSTERS_0_DISABLELOGDIRSCOLLECTION=true \
        -e KAFKA_CLUSTERS_0_PROPERTIES_ENABLE_IDEMPOTENCE='false' \
        -e KAFKA_CLUSTERS_0_PROPERTIES_SECURITY_PROTOCOL=SASL_SSL \
        -e KAFKA_CLUSTERS_0_PROPERTIES_SASL_MECHANISM=PLAIN \
        -e KAFKA_CLUSTERS_0_PROPERTIES_SASL_JAAS_CONFIG='org.apache.kafka.common.security.plain.PlainLoginModule required username="xxx/xxx/xxx" password="xxx";' \
        provectuslabs/kafka-ui:latest

both latest and `master tags work for me.

Make sure that you using jaas config from oci console:

->

KAFKA-UI

[github-actions[bot]]

This issue has been automatically marked as stale because no requested feedback has been provided. It will be closed if no further activity occurs. Thank you for your contributions.