akhq icon indicating copy to clipboard operation
akhq copied to clipboard

Connection timeout to schema registry

Open mahauke opened this issue 3 years ago • 1 comments

Hi,

we are using AKHQ 0.20.0 as developer tool for Strimzi Kafka in Kubernetes with a Confluent Schema Registry 6.2.2 also hosted in Kubernetes.

We currently encounter issues using the "Schema Registry" part of AKHQ, since we get an empty error message in the UI after a long time of loading. I digged into the logs and found the following error is related to this behavior:

ERROR thread-243 o.a.c.ErrorController connect timed out
java.net.SocketTimeoutException: connect timed out
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.base/java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.base/java.net.SocksSocketImpl.connect(Unknown Source)
at java.base/java.net.Socket.connect(Unknown Source)
at java.base/sun.security.ssl.SSLSocketImpl.connect(Unknown Source)
at java.base/sun.net.NetworkClient.doConnect(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.http.HttpClient.openServer(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsClient.<init>(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsClient.New(Unknown Source)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.getNewHttpClient(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(Unknown Source)
at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(Unknown Source)
at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(Unknown Source)
at java.base/java.net.HttpURLConnection.getResponseCode(Unknown Source)
at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getResponseCode(Unknown Source)
at io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:277)
at io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:367)
at io.confluent.kafka.schemaregistry.client.rest.RestService.getAllSubjects(RestService.java:994)
at io.confluent.kafka.schemaregistry.client.rest.RestService.getAllSubjects(RestService.java:979)
at org.akhq.repositories.SchemaRegistryRepository.all(SchemaRegistryRepository.java:90)
at org.akhq.repositories.SchemaRegistryRepository.list(SchemaRegistryRepository.java:42)
at org.akhq.controllers.SchemaController.list(SchemaController.java:51)
at org.akhq.controllers.$SchemaController$Definition$Exec.dispatch(Unknown Source)
at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:351)
at io.micronaut.context.DefaultBeanContext$4.invoke(DefaultBeanContext.java:583)
at io.micronaut.web.router.AbstractRouteMatch.execute(AbstractRouteMatch.java:303)
at io.micronaut.web.router.RouteMatch.execute(RouteMatch.java:111)
at io.micronaut.http.context.ServerRequestContext.with(ServerRequestContext.java:103)
at io.micronaut.http.server.RouteExecutor.lambda$executeRoute$14(RouteExecutor.java:656)
at reactor.core.publisher.FluxDeferContextual.subscribe(FluxDeferContextual.java:49)
at reactor.core.publisher.InternalFluxOperator.subscribe(InternalFluxOperator.java:62)
at reactor.core.publisher.FluxSubscribeOn$SubscribeOnSubscriber.run(FluxSubscribeOn.java:194)
at io.micronaut.reactive.reactor.instrument.ReactorInstrumentation.lambda$null$0(ReactorInstrumentation.java:62)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
at io.micronaut.scheduling.instrument.InvocationInstrumenterWrappedCallable.call(InvocationInstrumenterWrappedCallable.java:53)
at io.micrometer.core.instrument.composite.CompositeTimer.recordCallable(CompositeTimer.java:68)
at io.micrometer.core.instrument.Timer.lambda$wrap$1(Timer.java:171)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)

The schemaregistry seems to run fine, manual API calls and also consumers and producers of our applications are running fine. Thus, it seems to be either a network issue, a certificate issue (we are using an internal CA) or an AKHQ issue.

I already tested connecting to the registry using the specified connection URL of the schema registry from the config file, by using curl from the AKHQ container. I had to apply --insecure, since I cannot use the truststore of AKHQ with curl and the certificate is signed by an internal CA. Still, the correct Root CA is in the Truststore used by AKHQ. And connecting with --insecure worked.

Since I have no idea where to investigate further into this issue, my question is: How can I get more information on the error logged by AKHQ? Is there any way to find out whether this is a network issue, a certificate issue or an AKHQ issue?

Just as a note: It seemed to work fine at the beginning when no schemas had been stored in the Registry. At least, no error came up - as far as I remember, but I cannot reproduce now...

Any ideas or suggestions?

Thank you in advance Mario Hauke

mahauke avatar Jun 24 '22 12:06 mahauke

Hello, Same error for me : { "sequenceNumber":0, "timestamp":1749044243998, "nanoseconds":998412880, "level":"ERROR", "threadName":"io-executor-thread-1", "loggerName":"org.akhq.controllers.ErrorController", "context":{ "name":"default", "birthdate":1749043889431, "properties":{} }, "mdc": {}, "message":"Connect timed out", "throwable":{ "className":"java.net.SocketTimeoutException", "message":"Connect timed out", "stepArray":[ { "className":"sun.nio.ch.NioSocketImpl", "methodName":"timedFinishConnect", "fileName":"null", "lineNumber":-1 },{ "className":"sun.nio.ch.NioSocketImpl", "methodName":"connect", "fileName":"null", "lineNumber":-1 },{"className":"java.net.Socket","methodName":"connect","fileName":"null","lineNumber":-1},{"className":"sun.net.NetworkClient","methodName":"doConnect","fileName":"null","lineNumber":-1},{"className":"sun.net.www.http.HttpClient","methodName":"openServer","fileName":"null","lineNumber":-1},{"className":"sun.net.www.http.HttpClient$1","methodName":"run","fileName":"null","lineNumber":-1},{"className":"sun.net.www.http.HttpClient$1","methodName":"run","fileName":"null","lineNumber":-1},{"className":"java.security.AccessController","methodName":"doPrivileged","fileName":"null","lineNumber":-1},{"className":"sun.net.www.http.HttpClient","methodName":"privilegedOpenServer","fileName":"null","lineNumber":-1},{"className":"sun.net.www.http.HttpClient","methodName":"openServer","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.https.HttpsClient","methodName":"","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.https.HttpsClient","methodName":"New","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection","methodName":"getNewHttpClient","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.http.HttpURLConnection","methodName":"plainConnect0","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.http.HttpURLConnection","methodName":"plainConnect","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection","methodName":"connect","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.http.HttpURLConnection","methodName":"getInputStream0","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.http.HttpURLConnection","methodName":"getInputStream","fileName":"null","lineNumber":-1},{"className":"java.net.HttpURLConnection","methodName":"getResponseCode","fileName":"null","lineNumber":-1},{"className":"sun.net.www.protocol.https.HttpsURLConnectionImpl","methodName":"getResponseCode","fileName":"null","lineNumber":-1},{"className":"io.confluent.kafka.schemaregistry.client.rest.RestService","methodName":"sendHttpRequest","fileName":"RestService.java","lineNumber":312},{"className":"io.confluent.kafka.schemaregistry.client.rest.RestService","methodName":"httpRequest","fileName":"RestService.java","lineNumber":408},{"className":"io.confluent.kafka.schemaregistry.client.rest.RestService","methodName":"getAllSubjects","fileName":"RestService.java","lineNumber":1111},{"className":"io.confluent.kafka.schemaregistry.client.rest.RestService","methodName":"getAllSubjects","fileName":"RestService.java","lineNumber":1096},{"className":"org.akhq.repositories.SchemaRegistryRepository","methodName":"all","fileName":"SchemaRegistryRepository.java","lineNumber":104},{"className":"org.akhq.repositories.SchemaRegistryRepository","methodName":"list","fileName":"SchemaRegistryRepository.java","lineNumber":56},{"className":"org.akhq.controllers.SchemaController","methodName":"list","fileName":"SchemaController.java","lineNumber":73},{"className":"org.akhq.controllers.$SchemaController$Definition$Exec","methodName":"dispatch","fileName":"null","lineNumber":-1},{"className":"io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod","methodName":"invokeUnsafe","fileName":"AbstractExecutableMethodsDefinition.java","lineNumber":461},{"className":"io.micronaut.context.DefaultBeanContext$BeanContextUnsafeExecutionHandle","methodName":"invokeUnsafe","fileName":"DefaultBeanContext.java","lineNumber":4276},{"className":"io.micronaut.web.router.AbstractRouteMatch","methodName":"execute","fileName":"AbstractRouteMatch.java","lineNumber":236},{"className":"io.micronaut.http.server.RouteExecutor","methodName":"executeRouteAndConvertBody","fileName":"RouteExecutor.java","lineNumber":488},{"className":"io.micronaut.http.server.RouteExecutor","methodName":"lambda$callRoute$6","fileName":"RouteExecutor.java","lineNumber":465},{"className":"io.micronaut.core.execution.ExecutionFlow","methodName":"lambda$async$1","fileName":"ExecutionFlow.java","lineNumber":87},{"className":"io.micronaut.core.propagation.PropagatedContext","methodName":"lambda$wrap$3","fileName":"PropagatedContext.java","lineNumber":211},{"className":"io.micrometer.core.instrument.composite.CompositeTimer","methodName":"record","fileName":"CompositeTimer.java","lineNumber":141},{"className":"io.micrometer.core.instrument.Timer","methodName":"lambda$wrap$0","fileName":"Timer.java","lineNumber":193},{"className":"io.micronaut.core.propagation.PropagatedContext","methodName":"lambda$wrap$3","fileName":"PropagatedContext.java","lineNumber":211},{"className":"io.micrometer.core.instrument.composite.CompositeTimer","methodName":"record","fileName":"CompositeTimer.java","lineNumber":141},{"className":"io.micrometer.core.instrument.Timer","methodName":"lambda$wrap$0","fileName":"Timer.java","lineNumber":193},{"className":"java.util.concurrent.ThreadPoolExecutor","methodName":"runWorker","fileName":"null","lineNumber":-1 },{ "className":"java.util.concurrent.ThreadPoolExecutor$Worker", "methodName":"run", "fileName":"null", "lineNumber":-1 },{ "className":"java.lang.Thread","methodName":"run","fileName":"null","lineNumber":-1}]}} {"sequenceNumber":0,"timestamp":1749044244002,"nanoseconds":2253655,"level":"WARN","threadName":"io-executor-thread-1","loggerName":"org.akhq.log.access","context":{"name":"default","birthdate":1749043889431,"properties":{}},"mdc": {},"message":"[Date: 2025-06-04T13:35:23.884120054Z] [Duration: 120118 ms] [Url: GET /api/prd/schema] [Status: 500] [Ip: /99.99.99.99] [User: [email protected]]","throwable":null} {"sequenceNumber":0,"timestamp":1749044294458,"nanoseconds":458569679,"level":"INFO","threadName":"kafka-admin-client-thread | adminclient-1","loggerName":"org.apache.kafka.clients.NetworkClient","context":{"name":"default","birthdate":1749043889431,"properties":{}},"mdc": {},"message":"[AdminClient clientId=adminclient-1] Node -1 disconnected.","throwable":null}

I'm using a http proxy for micronaut http client. I don't need a proxy to access Schema Registry. From my AKHQ k8s pod i'm able to curl Schema registry with and without HTTP proxy. I have also an empty error message inside AKHQ interface when i try to list schemas.

timBorelle avatar Jun 06 '25 15:06 timBorelle