ClickHouse
problem with default http-client in jdbc-driver
Clickhouse JDBC Driver 0.6.1, 0.6.2 Clickhouse version 24.3
We migrate data from Greenplum to Clickhouse by PXF (jdbc clickhouse 0.6.1 with default http-client)
How to reproduce
This error appears, when we try to insert a lot of data from Greenplum to CH by External table (PXF) 390 millions rows. There are about Greenplum segments 190+ and about 190+ connections.
jdbc parameters to migrate: Driver com.clickhouse.jdbc.ClickHouseDriver jdbc.batch_size 10000 jdbc.fetch_size 10000 pool_enabled true pool_max_size 200 poll_connection_timeout 10min pool_idle_timeout 30sec pool_min_idle 0
This error was in pxf-logs from Greenplum segments, but no errors from Clickhouse logs.
Error writing request body to server\n\njava.sql.BatchUpdateException: Error writing request body to server\n\tat com.clickhouse.jdbc.SqlExceptionUtils.batchUpdateError(SqlExceptionUtils.java:107) ~[clickhouse-jdbc-0.6.1-http.jar:clickhouse-jdbc 0.6.1 (revision: 5be856e)]\n\tat com.clickhouse.jdbc.internal.InputBasedPreparedStatement.executeAny(InputBasedPreparedStatement.java:154) ~[clickhouse-jdbc-0.6.1-http.jar:clickhouse-jdbc 0.6.1 (revision: 5be856e)]\n\tat com.clickhouse.jdbc.internal.AbstractPreparedStatement.executeLargeBatch(AbstractPreparedStatement.java:85) ~[clickhouse-jdbc-0.6.1-http.jar:clickhouse-jdbc 0.6.1 (revision: 5be856e)]\n\tat com.clickhouse.jdbc.internal.ClickHouseStatementImpl.executeBatch(ClickHouseStatementImpl.java:770) ~[clickhouse-jdbc-0.6.1-http.jar:clickhouse-jdbc 0.6.1 (revision: 5be856e)]\n\tat com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128) ~[HikariCP-3.4.5.jar!/:?]\n\tat
Then we try to use apache-http client and there is not some errors and insert finished correctly. We used 5.3.1 Apache Http Client. Parameters of jdbc are the same.
hi, @vpvha! Thank you for reporting this! It is correct that Apache HTTP client fixed the issue for you? In general we recommend using Apache HTTP for production because this client has proven itself to be reliable.
What client version and name caused problems? Do you have a full stacktrace of the exception?
Thanks in advance!
Hi, @chernser! Yes, AHC fixed this for me, but I want use default http-client =)
Version of client in stacktrace, we used default http-client from jdbc-client-0.6.1 and the same error was in 0.6.2.
Full stacktrace:
Error writing request body to server\n\njava.sql.BatchUpdateException: Error writing request body to server\n\tat com.clickhouse.jdbc.SqlExceptionUtils.batchUpdateError(SqlExceptionUtils.java:107) ~[clickhouse-jdbc-0.6.1-http.jar:clickhouse-jdbc 0.6.1 (revision: 5be856e)]\n\tat com.clickhouse.jdbc.internal.InputBasedPreparedStatement.executeAny(InputBasedPreparedStatement.java:154) ~[clickhouse-jdbc-0.6.1-http.jar:clickhouse-jdbc 0.6.1 (revision: 5be856e)]\n\tat com.clickhouse.jdbc.internal.AbstractPreparedStatement.executeLargeBatch(AbstractPreparedStatement.java:85) ~[clickhouse-jdbc-0.6.1-http.jar:clickhouse-jdbc 0.6.1 (revision: 5be856e)]\n\tat com.clickhouse.jdbc.internal.ClickHouseStatementImpl.executeBatch(ClickHouseStatementImpl.java:770) ~[clickhouse-jdbc-0.6.1-http.jar:clickhouse-jdbc 0.6.1 (revision: 5be856e)]\n\tat com.zaxxer.hikari.pool.ProxyStatement.executeBatch(ProxyStatement.java:128) ~[HikariCP-3.4.5.jar!/:?]\n\tat com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeBatch(HikariProxyPreparedStatement.java) ~[HikariCP-3.4.5.jar!/:?]\n\tat org.greenplum.pxf.plugins.jdbc.writercallable.BatchWriterCallable.call(BatchWriterCallable.java:73) ~[pxf-jdbc-6.8.0.jar!/:?]\n\tat org.greenplum.pxf.plugins.jdbc.JdbcAccessor.writeNextObject(JdbcAccessor.java:269) ~[pxf-jdbc-6.8.0.jar!/:?]\n\tat org.greenplum.pxf.service.bridge.WriteBridge.setNext(WriteBridge.java:97) ~[classes!/:6.8.0]\n\tat org.greenplum.pxf.service.controller.WriteServiceImpl.readStream(WriteServiceImpl.java:67) ~[classes!/:6.8.0]\n\tat org.greenplum.pxf.service.controller.WriteServiceImpl.lambda$writeData$0(WriteServiceImpl.java:40) ~[classes!/:6.8.0]\n\tat java.security.AccessController.doPrivileged(Native Method) ~[?:?]\n\tat javax.security.auth.Subject.doAs(Subject.java:361) ~[?:?]\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1918) ~[hadoop-common-2.10.2.jar!/:?]\n\tat org.greenplum.pxf.service.security.BaseSecurityService.doAs(BaseSecurityService.java:122) ~[classes!/:6.8.0]\n\tat org.greenplum.pxf.service.controller.BaseServiceImpl.processData(BaseServiceImpl.java:74) ~[classes!/:6.8.0]\n\tat org.greenplum.pxf.service.controller.WriteServiceImpl.writeData(WriteServiceImpl.java:40) ~[classes!/:6.8.0]\n\tat org.greenplum.pxf.service.rest.PxfWriteResource.produceResponse(PxfWriteResource.java:52) ~[classes!/:6.8.0]\n\tat org.greenplum.pxf.service.rest.PxfWriteResource.produceResponse(PxfWriteResource.java:19) ~[classes!/:6.8.0]\n\tat org.greenplum.pxf.service.rest.PxfBaseResource.lambda$processRequest$0(PxfBaseResource.java:51) ~[classes!/:6.8.0]\n\tat org.greenplum.pxf.service.controller.PxfErrorReporter.invokeWithErrorHandling(PxfErrorReporter.java:26) [classes!/:6.8.0]\n\tat org.greenplum.pxf.service.rest.PxfBaseResource.processRequest(PxfBaseResource.java:48) [classes!/:6.8.0]\n\tat org.greenplum.pxf.service.rest.PxfWriteResource.stream(PxfWriteResource.java:47) [classes!/:6.8.0]\n\tat jdk.internal.reflect.GeneratedMethodAccessor64.invoke(Unknown Source) ~[?:?]\n\tat jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]\n\tat java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1072) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:965) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:555) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) [spring-webmvc-5.3.27.jar!/:5.3.27]\n\tat javax.servlet.http.HttpServlet.service(HttpServlet.java:623) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51) [tomcat-embed-websocket-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.greenplum.pxf.service.spring.PxfContextMdcLogEnhancerFilter.doFilterInternal(PxfContextMdcLogEnhancerFilter.java:41) [classes!/:6.8.0]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) [spring-boot-actuator-2.7.12.jar!/:2.7.12]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) [spring-web-5.3.27.jar!/:5.3.27]\n\tat org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:390) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:926) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1791) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) [tomcat-embed-core-9.0.75.jar!/:?]\n\tat java.lang.Thread.run(Thread.java:829) [?:?]\n"
Closing this out as Apache HTTP client solved the issue in V1 (and we also use the Apache client for V2)
