snowflake-jdbc icon indicating copy to clipboard operation
snowflake-jdbc copied to clipboard

Published 20 hours ago • verified publisher icon snowflakedb

SNOW-755767: snowflake-jdbc driver fails with JDK >= 16

Open sreev opened this issue 4 years ago • 32 comments

env: mac v11.6 intellij openjdk17 snowflake-driver v3.13.8 jar from maven repo

stack trace: Exception in thread "main" net.snowflake.client.jdbc.SnowflakeSQLLoggedException: JDBC driver internal error: Fail to retrieve row count for first arrow chunk: sun.misc.Unsafe or java.nio.DirectByteBuffer.<init>(long, int) not available. at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.setFirstChunkRowCountForArrow(SnowflakeResultSetSerializableV1.java:1061) at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.create(SnowflakeResultSetSerializableV1.java:550) at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.create(SnowflakeResultSetSerializableV1.java:467) at net.snowflake.client.core.SFResultSetFactory.getResultSet(SFResultSetFactory.java:29) at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:219) at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:134) at net.snowflake.client.core.SFStatement.execute(SFStatement.java:743) at net.snowflake.client.core.SFStatement.execute(SFStatement.java:639) at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:238) at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQuery(SnowflakeStatementV1.java:133)

workaround: Change jdk17 to jdk11.

sreev avatar Sep 28 '21 23:09 sreev

I have the same problem.

Ubuntu 20.04.3 LTS
openjdk 17 2021-09-14
OpenJDK Runtime Environment Temurin-17+35 (build 17+35)
OpenJDK 64-Bit Server VM Temurin-17+35 (build 17+35, mixed mode, sharing)

Downgrading to JDK 16 "fixes" the problem

fprochazka avatar Oct 14 '21 16:10 fprochazka

Adding the JVM argument: --add-opens java.base/java.nio=ALL-UNNAMED seems to solve the problem.

benveniste avatar Dec 01 '21 18:12 benveniste

Is there a solution without downgrading to JDK 16?

iamzafar avatar Jan 13 '22 22:01 iamzafar

@sfc-gh-kdama @sfc-gh-mknister Could you please look at this issue? It seems like #682 solves that. Releasing this would be really appreciated because our snowflake apps are failing with JDK17. Is there any chance to include this fix in the next following release?

iamzafar avatar Jan 21 '22 20:01 iamzafar

I also vote for fixing it. For me, the driver fails even with the add-opens flag.

dbschema-pro avatar Feb 03 '22 06:02 dbschema-pro

I wasn't able to create the ticket but found one that's already been reported #533

iamzafar avatar Feb 16 '22 20:02 iamzafar

It looks that ARROW-12747 was finally resolved three days ago in version 8.0.0. I was able to build snowflake-jdbc project with shaded arrow 8.0.0 and it became usable under Java 17. Still had to add --add-opens java.base/java.nio=ALL-UNNAMED, though. Let's hope that we'll get soon a new snowflake-jdbc:3.13.19 release with upgraded arrow that will be finally compatible with Java 17. =)

vkopichenko avatar May 13 '22 22:05 vkopichenko

Adding a VM argument is not a good way to do this. I'd use it if it's really necessary. Otherwise, I avoid using VM arguments in production during the runtime because VM arguments are a black box to me.

iamzafar avatar May 16 '22 20:05 iamzafar

The issue can be marked as resolved since 3.13.19 snowflake-jdbc driver version.

vkopichenko avatar May 31 '22 09:05 vkopichenko

I haven't tested, but are you sure @vkopichenko ? Snowflake 3.13.19 is using Arrow 7: https://github.com/snowflakedb/snowflake-jdbc/blob/v3.13.19/pom.xml#L40 But the fix is only included in Arrow 8: https://issues.apache.org/jira/browse/ARROW-12747

flamber avatar May 31 '22 10:05 flamber

Oh. It seems I got deceived by long-awaited eager expectations. Sorry for spammy false alarm. Will have to train attentiveness and patience a bit longer. )

vkopichenko avatar May 31 '22 11:05 vkopichenko

This should be addressed in #1017

sfc-gh-wfateem avatar May 31 '22 22:05 sfc-gh-wfateem

After upgrading to 3.13.20 we are still seeing errors with JDK 17

java.lang.ExceptionInInitializerError: null
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.UnsafeAllocationManager.<clinit>(UnsafeAllocationManager.java:27)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.DefaultAllocationManagerFactory.<clinit>(DefaultAllocationManagerFactory.java:26)
	at java.base/java.lang.Class.forName0(Native Method)
	at java.base/java.lang.Class.forName(Class.java:375)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.DefaultAllocationManagerOption.getFactory(DefaultAllocationManagerOption.java:111)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.DefaultAllocationManagerOption.getDefaultAllocationManagerFactory(DefaultAllocationManagerOption.java:101)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.BaseAllocator$Config.getAllocationManagerFactory(BaseAllocator.java:735)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.ImmutableConfig.access$801(ImmutableConfig.java:24)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.ImmutableConfig$InitShim.getAllocationManagerFactory(ImmutableConfig.java:83)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:47)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.ImmutableConfig.<init>(ImmutableConfig.java:24)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.ImmutableConfig$Builder.build(ImmutableConfig.java:485)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.BaseAllocator.<clinit>(BaseAllocator.java:63)
	at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.create(SnowflakeResultSetSerializableV1.java:560)
	at net.snowflake.client.jdbc.SnowflakeResultSetSerializableV1.create(SnowflakeResultSetSerializableV1.java:479)
	at net.snowflake.client.core.SFResultSetFactory.getResultSet(SFResultSetFactory.java:29)
	at net.snowflake.client.core.SFStatement.executeQueryInternal(SFStatement.java:220)
	at net.snowflake.client.core.SFStatement.executeQuery(SFStatement.java:135)
	at net.snowflake.client.core.SFStatement.execute(SFStatement.java:768)
	at net.snowflake.client.core.SFStatement.execute(SFStatement.java:678)
	at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:238)
	at net.snowflake.client.jdbc.SnowflakePreparedStatementV1.executeQuery(SnowflakePreparedStatementV1.java:117)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
	at io.opentracing.contrib.jdbc.TracingPreparedStatement.executeQuery(TracingPreparedStatement.java:62)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.opentracing.contrib.common.WrapperProxy$1.invoke(WrapperProxy.java:73)
	at jdk.proxy2/jdk.proxy2.$Proxy264.executeQuery(Unknown Source)
	at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:722)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:757)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:815)
	at com.reece.platform.products.pdw.repository.DataWarehouseRepository.getChangedProducts(DataWarehouseRepository.java:46)
	at com.reece.platform.products.pdw.repository.DataWarehouseRepository$$FastClassBySpringCGLIB$$36bc6300.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
	at com.reece.platform.products.pdw.repository.DataWarehouseRepository$$EnhancerBySpringCGLIB$$3c90c8c9.getChangedProducts(<generated>)
	at com.reece.platform.products.search.CreateIndexService.loadChangedProducts(CreateIndexService.java:145)
	at com.reece.platform.products.search.CreateIndexService.updateCurrentEngine(CreateIndexService.java:99)
	at com.reece.platform.products.controller.JobsController.updateIndex(JobsController.java:31)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:920)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:684)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.RuntimeException: Failed to initialize MemoryUtil.
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:139)
	... 108 common frames omitted
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field long java.nio.Buffer.address accessible: module java.base does not "opens java.nio" to unnamed module @5a4aa2f2
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
	at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
	at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
	at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
	at net.snowflake.client.jdbc.internal.apache.arrow.memory.util.MemoryUtil.<clinit>(MemoryUtil.java:87)
	... 108 common frames omitted

...

net.snowflake.client.jdbc.SnowflakeSQLException: JDBC driver internal error: exception creating result java.lang.ExceptionInInitializerError at net.snowflake.client.jdbc.internal.apache.arrow.memory.UnsafeAllocationManager.<clinit>(UnsafeAllocationManager.java:27).
	at net.snowflake.client.jdbc.SnowflakeStatementV1.executeQueryInternal(SnowflakeStatementV1.java:245)
	at net.snowflake.client.jdbc.SnowflakePreparedStatementV1.executeQuery(SnowflakePreparedStatementV1.java:117)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeQuery(ProxyPreparedStatement.java:52)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeQuery(HikariProxyPreparedStatement.java)
	at io.opentracing.contrib.jdbc.TracingPreparedStatement.executeQuery(TracingPreparedStatement.java:62)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at io.opentracing.contrib.common.WrapperProxy$1.invoke(WrapperProxy.java:73)
	at jdk.proxy2/jdk.proxy2.$Proxy264.executeQuery(Unknown Source)
	at org.springframework.jdbc.core.JdbcTemplate$1.doInPreparedStatement(JdbcTemplate.java:722)
	at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:651)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:713)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:744)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:757)
	at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:815)
	at com.reece.platform.products.pdw.repository.DataWarehouseRepository.getChangedProducts(DataWarehouseRepository.java:46)
	at com.reece.platform.products.pdw.repository.DataWarehouseRepository$$FastClassBySpringCGLIB$$36bc6300.invoke(<generated>)
	at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:137)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
	at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
	at com.reece.platform.products.pdw.repository.DataWarehouseRepository$$EnhancerBySpringCGLIB$$3c90c8c9.getChangedProducts(<generated>)
	at com.reece.platform.products.search.CreateIndexService.loadChangedProducts(CreateIndexService.java:145)
	at com.reece.platform.products.search.CreateIndexService.updateCurrentEngine(CreateIndexService.java:99)
	at com.reece.platform.products.controller.JobsController.updateIndex(JobsController.java:31)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
	at org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:920)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:684)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:764)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:227)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at io.opentracing.contrib.web.servlet.filter.TracingFilter.doFilter(TracingFilter.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:189)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:162)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:197)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:135)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:360)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:399)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.base/java.lang.Thread.run(Thread.java:833)

datanrd avatar Jul 08 '22 14:07 datanrd

Confirmed that the problem still occurs with Apache Arrow v8.0. We'll need to look into this and see what's happening here.

sfc-gh-wfateem avatar Jul 08 '22 15:07 sfc-gh-wfateem

Confirmed that the problem still occurs with Apache Arrow v8.0. We'll need to look into this and see what's happening here.

https://arrow.apache.org/release/9.0.0.html was released August 3rd does this correct the issue?

mdiskin avatar Aug 18 '22 13:08 mdiskin

@mdiskin it won't unfortunately.

The issue really requires all developers to plan to migrate to modules, otherwise, so long as any project (whether that's Arrow, Snowflake-JDBC, or Netty) uses reflection then you will always see these warnings/errors in JDK 9 and above. These problems manifested themselves as just warnings but then turned into errors once the JDK turned on strong encapsulation by default starting with JDK 16.

The issue described here is directly caused by Apache Arrow Memory, and the same code exists in v9.0 that leverages the sun.misc package.

Here's the output of the jdeps command when run against some of the classes involved in the issue in the Apache Arrow project:

❯ jdeps -jdkinternals ./memory/memory-unsafe/target/classes/org/apache/arrow/memory/UnsafeAllocationManager.class
UnsafeAllocationManager.class -> /Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home/jre/lib/rt.jar
   org.apache.arrow.memory.UnsafeAllocationManager (UnsafeAllocationManager.class)
      -> sun.misc.Unsafe                                    JDK internal API (rt.jar)

Warning: JDK internal APIs are unsupported and private to JDK implementation that are
subject to be removed or changed incompatibly and could break your application.
Please modify your code to eliminate dependency on any JDK internal APIs.
For the most recent update on JDK internal API replacements, please check:
https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool

JDK Internal API                         Suggested Replacement
----------------                         ---------------------
sun.misc.Unsafe                          See http://openjdk.java.net/jeps/260

...

❯ jdeps -jdkinternals ./memory/memory-core/target/classes/org/apache/arrow/memory/util/MemoryUtil.class
MemoryUtil.class -> /Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home/jre/lib/rt.jar
   org.apache.arrow.memory.util.MemoryUtil (MemoryUtil.class)
      -> sun.misc.Unsafe                                    JDK internal API (rt.jar)

Warning: JDK internal APIs are unsupported and private to JDK implementation that are
subject to be removed or changed incompatibly and could break your application.
Please modify your code to eliminate dependency on any JDK internal APIs.
For the most recent update on JDK internal API replacements, please check:
https://wiki.openjdk.java.net/display/JDK8/Java+Dependency+Analysis+Tool

JDK Internal API                         Suggested Replacement
----------------                         ---------------------
sun.misc.Unsafe                          See http://openjdk.java.net/jeps/260

So the real solution is to use the following guide to migrate all the projects: Migrating to JDK 9

As you can imagine, that will probably take time. I see Arrow started working on this. Even though the JIRA was closed as "incomplete," there is a PR that's still in progress: https://issues.apache.org/jira/browse/ARROW-16328

sfc-gh-wfateem avatar Aug 18 '22 14:08 sfc-gh-wfateem

I still face the same issue. JDK 17, Spring Boot 3 and Snowflake JDBC version 3.13.28. Thanks

LiamAttardcol avatar Mar 08 '23 16:03 LiamAttardcol

@LiamAttardcol FYI, I wrote this article to explain the nature of the problem. I welcome any feedback if there's something more I can add there to improve it: JDBC Driver Compatibility Issue With JDK 16 and Later

sfc-gh-wfateem avatar Mar 08 '23 19:03 sfc-gh-wfateem

Similar Exception when switching from jdk 15 to jdk 16

sfc-gh-igarish avatar Mar 09 '23 00:03 sfc-gh-igarish

Is there any update on what the latest for this is? This is an impediment to us being able to update our apps to Spring Boot 3 (which requires Java 17), so would be keen to hear some sort of time scale on a fix. Many thanks!

kyrielia avatar May 31 '23 07:05 kyrielia

Hi @kyrielia,

The issue isn't something we can simply fix on our own. It requires Apache Arrow to change their code in order to use public APIs as opposed to resorting to deep reflection to access private APIs in the JDK libraries.

Please refer to the following article: JDBC Driver Compatibility Issue With JDK 16 and Later

sfc-gh-wfateem avatar May 31 '23 13:05 sfc-gh-wfateem

Just noting another workaround (not ideal): https://github.com/snowflakedb/snowflake-jdbc/issues/533 ALTER SESSION SET JDBC_QUERY_RESULT_FORMAT='JSON' Or in the jdbc url: &JDBC_QUERY_RESULT_FORMAT=JSON

This is what we've used for a few months without issue.

mcouillard avatar May 31 '23 13:05 mcouillard

@sfc-gh-wfateem Ah my mistake, I misunderstood what the current status is from the messages in the thread so far, so thanks for clarifying

Is there an Apache Arrow issue to track here?

kyrielia avatar May 31 '23 13:05 kyrielia

Just noting another workaround (not ideal): #533 ALTER SESSION SET JDBC_QUERY_RESULT_FORMAT='JSON' Or in the jdbc url: &JDBC_QUERY_RESULT_FORMAT=JSON

This is what we've used for a few months without issue.

I've added this to JPA query ALTER SESSION SET JDBC_QUERY_RESULT_FORMAT='JSON' the problem is resolved but still, I'm getting previous exceptions on the console. Do you know if this is the current behavior???

arunjain501 avatar Jun 03 '23 05:06 arunjain501

@arunjain501 If you're 100% sure you're experiencing the same issue that's being discussed here, then it's highly likely that your application is creating multiple sessions and you've only altered one. Try running the same query in the Snowflake UI and just alter your user instead: ALTER USER SET JDBC_QUERY_RESULT_FORMAT='JSON' If you're still having issues, then please open a support ticket with us and we'll be happy to help.

sfc-gh-wfateem avatar Jun 03 '23 15:06 sfc-gh-wfateem

@kyrielia The issue was attempted to be addressed by Arrow in ARROW-16328 with PR 13072 . However, it looks like that was abandoned. I looked at the draft code changes and it still requires a bit more work. The easy part is defining a module and stating what modules you require, but the real issue is that the JDK's module descriptor for java.base simply exports java.nio and doesn't open it up to other modules. That means you simply can't use deep reflection anymore to access private members of classes in the java.nio package like Buffer and all its subclasses (ByteBuffer, CharBuffer, etc...). That essentially means Arrow has to refactor all its code to use compatible APIs to manage its buffers in memory, so it's not a small change.

sfc-gh-wfateem avatar Jun 05 '23 13:06 sfc-gh-wfateem

I heard second-hand from a teammate that the following JVM flags fixed this issue (unsure which subset is necessary):

                --add-opens=java.base/jdk.internal.misc=ALL-UNNAMED
                --add-opens=java.base/sun.nio.ch=ALL-UNNAMED
                --add-opens=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED
                --add-opens=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED
                --add-opens=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED
                --add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED
                --add-opens=java.base/java.io=ALL-UNNAMED
                --add-opens=java.base/java.nio=ALL-UNNAMED
                --add-opens=java.base/java.util=ALL-UNNAMED
                --add-opens=java.base/java.lang=ALL-UNNAMED
                --add-opens=java.base/java.lang.invoke=ALL-UNNAMED
                --add-opens=java.base/java.math=ALL-UNNAMED
                --add-opens=java.base/java.util.concurrent.atomic=ALL-UNNAMED
                --add-opens=java.sql/java.sql=ALL-UNNAMED

That's all the info I have. Hope it's useful.

spencerwilson avatar Oct 13 '23 15:10 spencerwilson

If you alter your result format to JSON as a workaround, as I had been doing, be careful that you don't need the additional precision for double numbers. I hit this: #898

jdimeo avatar Nov 03 '23 18:11 jdimeo

Just wanted to update others affected: Arrow won't work on the ticket that unblocks JDK 17 support until at least Jun 2024: https://github.com/apache/arrow/pull/13072#issuecomment-1814991745

mmendonca2 avatar Nov 16 '23 19:11 mmendonca2

&JDBC_QUERY_RESULT_FORMAT=JSON

This worked for me.

hkasi avatar Nov 23 '23 00:11 hkasi