trino-getting-started Failed connecting to Hive metastore

I am following the README instructions for Hive connector. I am getting Query 20220527_145544_00002_b3ih3 failed: Failed connecting to Hive metastore: [hive:9083] error for the latest Trino version 380.

May 27 '22 15:05 aksakalli

hi @aksakalli, I don't have a solution to offer you right this minute - but I potentially have a workaround if you would like to continue with the tutorial. I have been playing with this repo recently myself (Mac, M1-chip) and noticed the same issue intermittently. I started watching for all the containers to hit a point of stability in docker desktop before running my docker container exec -it trino-minio-trino-coordinator-1 trino command. Sometimes I would have to restart both the trino-minio-hive-metastore-1 or the trino-minio-mariadb1 containers. This fixed my issue.

May 27 '22 16:05 monimiller

@aksakalli , let me know if that is your issue, otherwise, I do notice the error message saying you're trying to connect to hive:9083 as the url. The hostname we assign in the docker image on the local network is hive-metastore, so likely there may be a setting in the minio catalog you changed that is pointing it to the wrong url?

May 27 '22 21:05 bitsondatadev

@aksakalli , let me know if that is your issue, otherwise, I do notice the error message saying you're trying to connect to hive:9083 as the url. The hostname we assign in the docker image on the local network is hive-metastore, so likely there may be a setting in the minio catalog you changed that is pointing it to the wrong url?

sorry I was wrong, I adopted it from a different log output but it was the same for the host hive-metastore. Also the latest version is 382 but the problem persists for 381 and 380.

hi @aksakalli, I don't have a solution to offer you right this minute - but I potentially have a workaround if you would like to continue with the tutorial. I have been playing with this repo recently myself (Mac, M1-chip) and noticed the same issue intermittently. I started watching for all the containers to hit a point of stability in docker desktop before running my docker container exec -it trino-minio-trino-coordinator-1 trino command. Sometimes I would have to restart both the trino-minio-hive-metastore-1 or the trino-minio-mariadb1 containers. This fixed my issue.

indeed, I was trying it on Apple M1-chip. I will try your workaround.

May 28 '22 00:05 aksakalli

Thanks @aksakalli, let us know if the workaround works for you. We may need to find something more permanent for M1 chip users.

May 29 '22 16:05 bitsondatadev

Hey @bitsondatadev,

I actually did a PR on your repo https://github.com/bitsondatadev/hive-metastore/pull/1, then simply change the image to ghcr.io/bitsondatadev/hive-metastore:latest.

Jun 02 '22 16:06 mdesmet

I am using MacBook Pro with M1 chip. I followed the steps based on https://github.com/bitsondatadev/trino-getting-started/tree/main/hive/trino-minio

I met same error

➜ docker compose up -d

[+] Running 6/6
 ✔ Network trino-minio_trino-network                                                                                                                             Created                                                                                                                                                 0.0s
 ✔ Container trino-minio-trino-coordinator-1                                                                                                                     Started                                                                                                                                                 0.0s
 ✔ Container trino-minio-mariadb-1                                                                                                                               Started                                                                                                                                                 0.0s
 ✔ Container minio                                                                                                                                               Started                                                                                                                                                 0.0s
 ✔ Container trino-minio-hive-metastore-1                                                                                                                        Started                                                                                                                                                 0.0s
 ! hive-metastore The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                                                                                                                         0.0s

Created tiny bucket.

➜ docker container exec -it trino-minio-trino-coordinator-1 trino
trino> SHOW CATALOGS;
    ->
 Catalog
---------
 minio
 system
 tpcds
 tpch
(4 rows)

Query 20230831_072414_00000_3ngfk, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0.63 [0 rows, 0B] [0 rows/s, 0B/s]

When I run

trino> CREATE SCHEMA minio.tiny
    -> WITH (location = 's3a://tiny/');
Query 20230831_072609_00001_3ngfk failed: Failed connecting to Hive metastore: [hive-metastore:9083]

My hive-metastore will crash with error:

Click to expand!

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2023-08-31 07:29:03,923 main INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
MetaException(message:Error creating transactional connection factory)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8541)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8536)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8806)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8723)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: MetaException(message:Error creating transactional connection factory)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
	... 11 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:671)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:830)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1975)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1970)
	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1177)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:814)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:702)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:647)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:690)
	at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:480)
	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:417)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:372)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:705)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:683)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:677)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:760)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:525)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	... 13 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
	at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
	at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
	at sun.reflect.GeneratedConstructorAccessor79.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
	at org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
	at org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
	... 43 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error : Failed to initialize pool: Too many connections
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:232)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
	... 59 more
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Too many connections
	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:544)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:536)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
	at org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
	... 61 more
Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
	... 65 more
Exception in thread "main" MetaException(message:Error creating transactional connection factory)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8541)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8536)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8806)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8723)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: MetaException(message:Error creating transactional connection factory)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
	... 11 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:671)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:830)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1975)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1970)
	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1177)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:814)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:702)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:647)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:690)
	at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:480)
	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:417)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:372)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:705)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:683)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:677)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:760)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:525)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	... 13 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
	at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
	at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
	at sun.reflect.GeneratedConstructorAccessor79.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
	at org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
	at org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
	... 43 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error : Failed to initialize pool: Too many connections
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:232)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
	... 59 more
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Too many connections
	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:544)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:536)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
	at org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
	... 61 more
Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
	... 65 more
2023-08-31 07:29:40,989 shutdown-hook-0 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
2023-08-31 07:29:40,997 shutdown-hook-0 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
2023-08-31 07:29:41,019 shutdown-hook-0 WARN Unable to register Log4j shutdown hook because JVM is shutting down. Using SimpleLogger

I tried to update this line with ghcr.io/bitsondatadev/hive-metastore:latest from @mdesmet

and run

docker-compose down
docker compose up -d

However, this time hive-metastore simply won't start no matter how many times I manually restart it. 😭

Aug 31 '23 07:08 hongbo-miao

I am using MacBook Pro with M1 chip. I followed the steps based on https://github.com/bitsondatadev/trino-getting-started/tree/main/hive/trino-minio

I met same error

➜ docker compose up -d

[+] Running 6/6
 ✔ Network trino-minio_trino-network                                                                                                                             Created                                                                                                                                                 0.0s
 ✔ Container trino-minio-trino-coordinator-1                                                                                                                     Started                                                                                                                                                 0.0s
 ✔ Container trino-minio-mariadb-1                                                                                                                               Started                                                                                                                                                 0.0s
 ✔ Container minio                                                                                                                                               Started                                                                                                                                                 0.0s
 ✔ Container trino-minio-hive-metastore-1                                                                                                                        Started                                                                                                                                                 0.0s
 ! hive-metastore The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested                                                                                                                                                         0.0s

Created tiny bucket.

➜ docker container exec -it trino-minio-trino-coordinator-1 trino
trino> SHOW CATALOGS;
    ->
 Catalog
---------
 minio
 system
 tpcds
 tpch
(4 rows)

Query 20230831_072414_00000_3ngfk, FINISHED, 1 node
Splits: 19 total, 19 done (100.00%)
0.63 [0 rows, 0B] [0 rows/s, 0B/s]

When I run

trino> CREATE SCHEMA minio.tiny
    -> WITH (location = 's3a://tiny/');
Query 20230831_072609_00001_3ngfk failed: Failed connecting to Hive metastore: [hive-metastore:9083]

My hive-metastore will crash with error:

Click to expand!

SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
2023-08-31 07:29:03,923 main INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
MetaException(message:Error creating transactional connection factory)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8541)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8536)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8806)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8723)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: MetaException(message:Error creating transactional connection factory)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
	... 11 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:671)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:830)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1975)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1970)
	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1177)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:814)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:702)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:647)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:690)
	at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:480)
	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:417)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:372)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:705)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:683)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:677)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:760)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:525)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	... 13 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
	at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
	at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
	at sun.reflect.GeneratedConstructorAccessor79.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
	at org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
	at org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
	... 43 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error : Failed to initialize pool: Too many connections
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:232)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
	... 59 more
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Too many connections
	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:544)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:536)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
	at org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
	... 61 more
Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
	... 65 more
Exception in thread "main" MetaException(message:Error creating transactional connection factory)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8541)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8536)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8806)
	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8723)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: MetaException(message:Error creating transactional connection factory)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
	... 11 more
Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
NestedThrowables:
java.lang.reflect.InvocationTargetException
	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:671)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:830)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1975)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1970)
	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1177)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:814)
	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:702)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:647)
	at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:690)
	at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:480)
	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:417)
	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:372)
	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:705)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:683)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:677)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:760)
	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:525)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
	... 13 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.GeneratedConstructorAccessor21.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
	at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
	at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
	at sun.reflect.GeneratedConstructorAccessor79.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
	at org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
	at org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
	... 43 more
Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error : Failed to initialize pool: Too many connections
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:232)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
	... 59 more
Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Too many connections
	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:544)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:536)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
	at org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
	... 61 more
Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
	... 65 more
2023-08-31 07:29:40,989 shutdown-hook-0 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
2023-08-31 07:29:40,997 shutdown-hook-0 INFO Log4j appears to be running in a Servlet environment, but there's no log4j-web module available. If you want better web container support, please add the log4j-web JAR to your web archive or server lib directory.
2023-08-31 07:29:41,019 shutdown-hook-0 WARN Unable to register Log4j shutdown hook because JVM is shutting down. Using SimpleLogger

I tried to update this line with ghcr.io/bitsondatadev/hive-metastore:latest from @mdesmet

and run

docker-compose down
docker compose up -d

However, this time hive-metastore simply won't start no matter how many times I manually restart it. 😭

I met the same questions. I found it caused by mariadb verison. You can set mariadb verison like 10.6.9 (before this repo date) If not OK, you can set max_connections and max_user_connections.

Oct 03 '23 02:10 Antony-Jia

This was related to an issue in mariadb, as of this commit we are using a version where this shouldn't continue causing any issues. Please reopen if it does.

Oct 16 '23 19:10 bitsondatadev

trino-getting-started trino-getting-started copied to clipboard

Failed connecting to Hive metastore

trino-getting-started
trino-getting-started copied to clipboard