Failing trino-iceberg-minio setup

[tschaub]

Running docker compose up with the trino-iceberg-minio example fails with things like this in the logs:

hive-metastore-1     | MetaException(message:Error creating transactional connection factory)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8541)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8536)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8806)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8723)
hive-metastore-1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
hive-metastore-1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
hive-metastore-1     | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hive-metastore-1     | 	at java.lang.reflect.Method.invoke(Method.java:498)
hive-metastore-1     | 	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
hive-metastore-1     | 	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
hive-metastore-1     | Caused by: MetaException(message:Error creating transactional connection factory)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
hive-metastore-1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
hive-metastore-1     | 	... 11 more

In trying to work around this, I have tried to modify the setup to use a JDBC catalog. See https://github.com/tschaub/trino-example.

[edkry]

Having the same error on Delta Lake setup

[iuliu-b]

Same error on hive/trino-minio, full stacktrace:

hive-metastore_1     | MetaException(message:Error creating transactional connection factory)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:84)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:93)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8541)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:8536)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:8806)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:8723)
hive-metastore_1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
hive-metastore_1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
hive-metastore_1     | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hive-metastore_1     | 	at java.lang.reflect.Method.invoke(Method.java:498)
hive-metastore_1     | 	at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
hive-metastore_1     | 	at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
hive-metastore_1     | Caused by: MetaException(message:Error creating transactional connection factory)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:208)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.<init>(RetryingHMSHandler.java:80)
hive-metastore_1     | 	... 11 more
hive-metastore_1     | Caused by: javax.jdo.JDOFatalInternalException: Error creating transactional connection factory
hive-metastore_1     | NestedThrowables:
hive-metastore_1     | java.lang.reflect.InvocationTargetException
hive-metastore_1     | 	at org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:671)
hive-metastore_1     | 	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:830)
hive-metastore_1     | 	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.createPersistenceManagerFactory(JDOPersistenceManagerFactory.java:334)
hive-metastore_1     | 	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.getPersistenceManagerFactory(JDOPersistenceManagerFactory.java:213)
hive-metastore_1     | 	at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
hive-metastore_1     | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hive-metastore_1     | 	at java.lang.reflect.Method.invoke(Method.java:498)
hive-metastore_1     | 	at javax.jdo.JDOHelper$16.run(JDOHelper.java:1975)
hive-metastore_1     | 	at java.security.AccessController.doPrivileged(Native Method)
hive-metastore_1     | 	at javax.jdo.JDOHelper.invoke(JDOHelper.java:1970)
hive-metastore_1     | 	at javax.jdo.JDOHelper.invokeGetPersistenceManagerFactoryOnImplementation(JDOHelper.java:1177)
hive-metastore_1     | 	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:814)
hive-metastore_1     | 	at javax.jdo.JDOHelper.getPersistenceManagerFactory(JDOHelper.java:702)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.getPMF(ObjectStore.java:647)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.getPersistenceManager(ObjectStore.java:690)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.initializeHelper(ObjectStore.java:480)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.initialize(ObjectStore.java:417)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.ObjectStore.setConf(ObjectStore.java:372)
hive-metastore_1     | 	at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:77)
hive-metastore_1     | 	at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:137)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RawStoreProxy.<init>(RawStoreProxy.java:59)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RawStoreProxy.getProxy(RawStoreProxy.java:67)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.newRawStoreForConf(HiveMetaStore.java:705)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMSForConf(HiveMetaStore.java:683)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.getMS(HiveMetaStore.java:677)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.createDefaultDB(HiveMetaStore.java:760)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:525)
hive-metastore_1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
hive-metastore_1     | 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
hive-metastore_1     | 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
hive-metastore_1     | 	at java.lang.reflect.Method.invoke(Method.java:498)
hive-metastore_1     | 	at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
hive-metastore_1     | 	... 13 more
hive-metastore_1     | Caused by: java.lang.reflect.InvocationTargetException
hive-metastore_1     | 	at sun.reflect.GeneratedConstructorAccessor20.newInstance(Unknown Source)
hive-metastore_1     | 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
hive-metastore_1     | 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
hive-metastore_1     | 	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
hive-metastore_1     | 	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:330)
hive-metastore_1     | 	at org.datanucleus.store.AbstractStoreManager.registerConnectionFactory(AbstractStoreManager.java:203)
hive-metastore_1     | 	at org.datanucleus.store.AbstractStoreManager.<init>(AbstractStoreManager.java:162)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.RDBMSStoreManager.<init>(RDBMSStoreManager.java:285)
hive-metastore_1     | 	at sun.reflect.GeneratedConstructorAccessor78.newInstance(Unknown Source)
hive-metastore_1     | 	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
hive-metastore_1     | 	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
hive-metastore_1     | 	at org.datanucleus.plugin.NonManagedPluginRegistry.createExecutableExtension(NonManagedPluginRegistry.java:606)
hive-metastore_1     | 	at org.datanucleus.plugin.PluginManager.createExecutableExtension(PluginManager.java:301)
hive-metastore_1     | 	at org.datanucleus.NucleusContextHelper.createStoreManagerForProperties(NucleusContextHelper.java:133)
hive-metastore_1     | 	at org.datanucleus.PersistenceNucleusContextImpl.initialise(PersistenceNucleusContextImpl.java:422)
hive-metastore_1     | 	at org.datanucleus.api.jdo.JDOPersistenceManagerFactory.freezeConfiguration(JDOPersistenceManagerFactory.java:817)
hive-metastore_1     | 	... 43 more
hive-metastore_1     | Caused by: org.datanucleus.exceptions.NucleusException: Attempt to invoke the "HikariCP" plugin to create a ConnectionPool gave an error : Failed to initialize pool: Too many connections
hive-metastore_1     | 	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:232)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.initialiseDataSources(ConnectionFactoryImpl.java:117)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.<init>(ConnectionFactoryImpl.java:82)
hive-metastore_1     | 	... 59 more
hive-metastore_1     | Caused by: com.zaxxer.hikari.pool.HikariPool$PoolInitializationException: Failed to initialize pool: Too many connections
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.throwPoolInitializationException(HikariPool.java:544)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:536)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112)
hive-metastore_1     | 	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:72)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.connectionpool.HikariCPConnectionPoolFactory.createConnectionPool(HikariCPConnectionPoolFactory.java:176)
hive-metastore_1     | 	at org.datanucleus.store.rdbms.ConnectionFactoryImpl.generateDataSources(ConnectionFactoryImpl.java:213)
hive-metastore_1     | 	... 61 more
hive-metastore_1     | Caused by: java.sql.SQLNonTransientConnectionException: Too many connections
hive-metastore_1     | 	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110)
hive-metastore_1     | 	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
hive-metastore_1     | 	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
hive-metastore_1     | 	at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:836)
hive-metastore_1     | 	at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:456)
hive-metastore_1     | 	at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:246)
hive-metastore_1     | 	at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:197)
hive-metastore_1     | 	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:95)
hive-metastore_1     | 	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:101)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.PoolBase.newConnection(PoolBase.java:356)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.PoolBase.newPoolEntry(PoolBase.java:199)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.createPoolEntry(HikariPool.java:444)
hive-metastore_1     | 	at com.zaxxer.hikari.pool.HikariPool.checkFailFast(HikariPool.java:515)
hive-metastore_1     | 	... 65 more

[iuliu-b]

This has not fixed trino-iceberg-minio for me (I realize that it should have no impact on trino-minio)

[tschaub]

@bitsondatadev - I should have been clearer, the stack trace above came after I had already removed the -XX:-UseBiasedLocking config (as you did in 5414225).

If others are looking for a Docker compose setup with Trino, Iceberg, and MinIO, here is what worked for me: https://github.com/tschaub/trino-example/blob/main/docker-compose.yml