Iceberg Rest Unable to list tables for Hive Metastore with mixed Hive and Iceberg

[metalshanked]

Version

main branch

Describe what's wrong

I am using Trino to connect to Gravitino Iceberg Rest Server backed by a Hive Metastore. The hive metastore can have native hive and iceberg tables both.

Below is the error in Trino and similar error in Gravitino logs too " Unable to process: hive is not support view"

Error message and/or stacktrace

Caused by: org.apache.iceberg.exceptions.RESTException: Unable to process: hive is not support view
	at org.apache.iceberg.rest.ErrorHandlers$DefaultErrorHandler.accept(ErrorHandlers.java:224)
	at org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:180)
	at org.apache.iceberg.rest.ErrorHandlers$NamespaceErrorHandler.accept(ErrorHandlers.java:166)
	at org.apache.iceberg.rest.HTTPClient.throwFailure(HTTPClient.java:211)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:323)
	at org.apache.iceberg.rest.HTTPClient.execute(HTTPClient.java:262)
	at org.apache.iceberg.rest.HTTPClient.get(HTTPClient.java:358)
	at org.apache.iceberg.rest.RESTClient.get(RESTClient.java:96)
	at org.apache.iceberg.rest.RESTSessionCatalog.listViews(RESTSessionCatalog.java:1133)
	at io.trino.plugin.iceberg.catalog.rest.TrinoRestCatalog.lambda$listTables$3(TrinoRestCatalog.java:218)
	at io.trino.plugin.iceberg.catalog.rest.TrinoRestCatalog.listTableIdentifiers(TrinoRestCatalog.java:243)
	... 49 more

How to reproduce

0.8.0

Additional context

No response

[FANNG1]

Seems Trino treat both Iceberg table and view to Trino tables. Hive catalog doesn't support view operations until 1.7.0 in https://github.com/apache/iceberg/pull/9852, while Gravitino still uses Iceberg 1.5.2. There may be two solutions:

1. return an empty list when listing view for the catalog doesn't support view operations.
2. upgrade Iceberg lib to 1.7.x, there maybe JDK compatibility problem, tracked in https://github.com/apache/gravitino/issues/6337

[FANNG1]

For solution one, suggest solution in IcebergCatalogWrapper, and welcome contribute.

  public ListTablesResponse listView(Namespace namespace) {
    if (!catalog instanceof ViewCatalog) {
       // return empty table list
    }
    return CatalogHandlers.listViews(getViewCatalog(), namespace);
  }

[FANNG1]

another more general way is to export Iceberg server capabilities in Iceberg REST config endpoint, but this works in Iceberg 1.7 in https://github.com/apache/iceberg/pull/10929.

[metalshanked]

Thanks for the info. Seems like path to iceberg 1.7 is best for a long term approach. Somehow, if Gravitino needs to update JDK and make it work.

[FANNG1]

Thanks for the info. Seems like path to iceberg 1.7 is best for a long term approach. Somehow, if Gravitino needs to update JDK and make it work.

Yes, this means Gravitino drops the support for JDK 8.

[FANNG1]

@metalshanked , should you share the Trino version and Iceberg version you are using? Iceberg library does some compatibility works to avoid the unexcepted exceptions when listing views.

[metalshanked]

Hi @FANNG1 . I was using the latest Trino version (469). I also tried with older version (459). I am using the Trino official Docker image.

Trino currently uses 1.7.1 but will up upgraded to 1.8 in the near future looks like

https://github.com/trinodb/trino/blob/3b53eee3c4cef19ea7c0aa8a20ca5987a7cd313e/pom.xml#L201

Future --> https://github.com/trinodb/trino/pull/24713

[metalshanked]

H @FANNG1 . trino added a new property 471 which I used iceberg.rest-catalog.view-endpoints-enabled=false

Now I get below error

2025-02-24 12:45:59.141 WARN [iceberg-rest-24] [org.apache.gravitino.iceberg.service.IcebergExceptionMapper.toResponse(IcebergExceptionMapper.java:78)] - Iceberg REST server unexpected exception:
java.lang.RuntimeException: Failed to list all tables under namespace myschema
        at org.apache.iceberg.hive.HiveCatalog.listTables(HiveCatalog.java:158) ~[iceberg-hive-metastore-1.5.2.jar:?]
        at org.apache.iceberg.rest.CatalogHandlers.listTables(CatalogHandlers.java:173) ~[iceberg-core-1.5.2.jar:?]
        at org.apache.gravitino.iceberg.common.ops.IcebergCatalogWrapper.listTable(IcebergCatalogWrapper.java:189) ~[gravitino-iceberg-common-0.8.0-incubating.jar:?]
        at org.apache.gravitino.iceberg.service.dispatcher.IcebergTableOperationExecutor.listTable(IcebergTableOperationExecutor.java:84) ~[gravitino-iceberg-rest-server-0.8.0-incubating.jar:?]
        at org.apache.gravitino.iceberg.service.dispatcher.IcebergTableEventDispatcher.listTable(IcebergTableEventDispatcher.java:170) ~[gravitino-iceberg-rest-server-0.8.0-incubating.jar:?]
        at org.apache.gravitino.iceberg.service.rest.IcebergTableOperations.listTable(IcebergTableOperations.java:96) ~[gravitino-iceberg-rest-server-0.8.0-incubating.jar:?]
        at jdk.internal.reflect.GeneratedMethodAccessor15.invoke(Unknown Source) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265) ~[jersey-common-2.41.jar:?]
        at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684) ~[jersey-server-2.41.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:358) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:311) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:205) ~[jersey-container-servlet-core-2.41.jar:?]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1656) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.apache.gravitino.server.authentication.AuthenticationFilter.doFilter(AuthenticationFilter.java:86) ~[gravitino-server-common-0.8.0-incubating.jar:?]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]
        at java.base/java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: org.apache.thrift.TApplicationException: Internal error processing get_table_objects_by_name
        at org.apache.thrift.TApplicationException.read(TApplicationException.java:111) ~[libthrift-0.9.3.jar:0.9.3]
        at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:79) ~[libthrift-0.9.3.jar:0.9.3]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table_objects_by_name(ThriftHiveMetastore.java:1544) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table_objects_by_name(ThriftHiveMetastore.java:1530) ~[hive-metastore-2.3.9.jar:2.3.9]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableObjectsByName(HiveMetaStoreClient.java:1363) ~[hive-metastore-2.3.9.jar:2.3.9]
        at jdk.internal.reflect.GeneratedMethodAccessor17.invoke(Unknown Source) ~[?:?]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
        at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[?:?]
        at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke(RetryingMetaStoreClient.java:169) ~[hive-metastore-2.3.9.jar:2.3.9]
        at jdk.proxy3/jdk.proxy3.$Proxy71.getTableObjectsByName(Unknown Source) ~[?:?]
        ... 61 more
2025-02-24 12:46:02.895 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:164)] - Total tree lock node count: 1
2025-02-24 12:46:02.895 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:115)] - Start to check the dead lock...
2025-02-24 12:46:02.896 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:117)] - Finish to check the dead lock...
2025-02-24 12:46:24.533 INFO [Gravitino-webserver-40] [org.apache.gravitino.server.web.rest.MetalakeOperations.listMetalakes(MetalakeOperations.java:84)] - Received list metalakes request.
2025-02-24 12:46:24.548 INFO [Gravitino-webserver-40] [org.apache.gravitino.server.web.rest.MetalakeOperations.lambda$listMetalakes$1(MetalakeOperations.java:94)] - List 0 metalakes in Gravitino
2025-02-24 12:47:02.895 INFO [tree-lock-cleaner-0] [org.apache.gravitino.lock.LockManager.lambda$startNodeCleaner$3(LockManager.java:164)] - Total tree lock node count: 1
2025-02-24 12:47:02.895 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:115)] - Start to check the dead lock...
2025-02-24 12:47:02.895 INFO [tree-lock-dead-lock-checker-0] [org.apache.gravitino.lock.LockManager.lambda$startDeadLockChecker$0(LockManager.java:117)] - Finish to check the dead lock...

[FANNG1]

looks like the Iceberg Hive catalog thrift client couldn't connect to the Hive metastore, some questions

1. is the hive metastore port correct?
2. which version is the hive metastore?
3. is hive metastore enabled security features?

[metalshanked]

Oh connection to metastore is fine (using Kerberized hive metastore) (it can read the schema (myschema) as shown in logs I pasted. Also, I see connected to hive metastore in logs @FANNG1 :- This happens only when queries are used to list tables in schema. Get schemas works fine.

[FANNG1]

Is there any useful message in Hivemetastore's log?