amoro Help me to connect s3 iceberg table with amoro ?

Give me detail steps

May 13 '25 19:05 omkenge

Thank you for following Apache Amoro. After the successful deploy amoro, you can enter the catalog setup and set warehouse address & endpoint & region. Later, I will supplement the documents and cases on how to use it in AWS S3.

May 14 '25 06:05 czy006

It is showing me like this

May 14 '25 06:05 omkenge

Metastore: Filesystem point to Hadoop Filesystem. If you require to manage the Iceberg tables on S3, please choise other catalogs such as GlueCatalog. I can show you how we create a glue catalog in Amoro:

AWS requires authentication information to access resources like S3 and Glue. Amoro offers flexible options for obtaining credentials:

Configure Access Key and Secret Key (AK/SK) directly on the UI.
By default, credentials are obtained from environment variables.
Additional configuration options compatible with Iceberg Catalog can be supplied through catalog properties, among others.

May 14 '25 09:05 xxubai

Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentityTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIVE_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]

Could you plz help me on this ? Detail

May 14 '25 11:05 omkenge

Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain

The error indicates that the Amoro management service did not obtain any AWS credentials, which means that no authentication information was configured in the application's global settings. The simplest way to resolve this is to provide the Access Key and Secret Key (AK/SK) properties when creating the catalog. A screenshot is shown below:

After created success. It will auto load iceberg tables in the catalog.

May 14 '25 11:05 xxubai

After adding AK/SK still tables are not visible .. Is it works for your setup ?

May 15 '25 07:05 omkenge

I'm facing the same issue and it appears that the AK/SK are only for S3. The stacktrace suggests that the connection to Glue is not using these AK/SK and instead it's falling back to AWS SDK defaults (i.e. try the Environment Vars first, etc).

Edit: confirming that Amoro is able to connect to Glue when I add the proper environment variables.

amoro-ams software.amazon.awssdk.core.exception.SdkClientException: Unable to load credentials from any of the providers in the chain AwsCredentialsProviderChain(credentialsProviders=[SystemPropertyCredentialsProvider(), EnvironmentVariableCredentialsProvider(), WebIdentit
yTokenCredentialsProvider(), ProfileCredentialsProvider(profileName=default, profileFile=ProfileFile(sections=[])), ContainerCredentialsProvider(), InstanceProfileCredentialsProvider()]) : [SystemPropertyCredentialsProvider(): Unable to load credentials from system setting
s. Access key must be specified either via environment variable (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., EnvironmentVariableCredentialsProvider(): Unable to load credentials from system settings. Access key must be specified either via environment variabl
e (AWS_ACCESS_KEY_ID) or system property (aws.accessKeyId)., WebIdentityTokenCredentialsProvider(): Either the environment variable AWS_WEB_IDENTITY_TOKEN_FILE or the javaproperty aws.webIdentityTokenFile must be set., ProfileCredentialsProvider(profileName=default, profil
eFile=ProfileFile(sections=[])): Profile file contained no credentials for profile 'default': ProfileFile(sections=[]), ContainerCredentialsProvider(): Cannot fetch credentials from container - neither AWS_CONTAINER_CREDENTIALS_FULL_URI or AWS_CONTAINER_CREDENTIALS_RELATIV
E_URI environment variables are set., InstanceProfileCredentialsProvider(): Failed to load credentials from IMDS.]
amoro-ams     at software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:111) ~[sdk-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:130) ~[auth-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45) ~[auth-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:128) ~[auth-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.auth.credentials.AwsCredentialsProvider.resolveIdentity(AwsCredentialsProvider.java:54) ~[auth-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.services.glue.auth.scheme.internal.GlueAuthSchemeInterceptor.lambda$trySelectAuthScheme$4(GlueAuthSchemeInterceptor.java:132) ~[glue-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.core.internal.util.MetricUtils.reportDuration(MetricUtils.java:77) ~[sdk-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.services.glue.auth.scheme.internal.GlueAuthSchemeInterceptor.trySelectAuthScheme(GlueAuthSchemeInterceptor.java:132) ~[glue-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.services.glue.auth.scheme.internal.GlueAuthSchemeInterceptor.selectAuthScheme(GlueAuthSchemeInterceptor.java:81) ~[glue-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.services.glue.auth.scheme.internal.GlueAuthSchemeInterceptor.beforeExecution(GlueAuthSchemeInterceptor.java:61) ~[glue-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.lambda$beforeExecution$1(ExecutionInterceptorChain.java:59) ~[sdk-core-2.24.12.jar:?]
amoro-ams     at java.util.ArrayList.forEach(ArrayList.java:1259) ~[?:1.8.0_452]
amoro-ams     at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.beforeExecution(ExecutionInterceptorChain.java:59) ~[sdk-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.runInitialInterceptors(AwsExecutionContextBuilder.java:239) ~[aws-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.awscore.internal.AwsExecutionContextBuilder.invokeInterceptorsAndCreateExecutionContext(AwsExecutionContextBuilder.java:130) ~[aws-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.invokeInterceptorsAndCreateExecutionContext(AwsSyncClientHandler.java:67) ~[aws-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:76) ~[sdk-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) ~[sdk-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) ~[sdk-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[sdk-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) ~[aws-core-2.24.12.jar:?]
amoro-ams     at software.amazon.awssdk.services.glue.DefaultGlueClient.getDatabases(DefaultGlueClient.java:6425) ~[glue-2.24.12.jar:?]
amoro-ams     at org.apache.iceberg.aws.glue.GlueCatalog.listNamespaces(GlueCatalog.java:483) ~[iceberg-aws-1.6.1.jar:?]
amoro-ams     at org.apache.iceberg.catalog.SupportsNamespaces.listNamespaces(SupportsNamespaces.java:74) ~[iceberg-api-1.6.1.jar:?]
amoro-ams     at org.apache.amoro.formats.iceberg.IcebergCatalog.lambda$listDatabases$1(IcebergCatalog.java:56) ~[amoro-format-iceberg-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:268) ~[amoro-common-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.table.TableMetaStore.doAs(TableMetaStore.java:241) ~[amoro-common-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.formats.iceberg.IcebergCatalog.listDatabases(IcebergCatalog.java:54) ~[amoro-format-iceberg-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.CommonUnifiedCatalog.listDatabases(CommonUnifiedCatalog.java:89) ~[amoro-common-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.server.catalog.ExternalCatalog.lambda$listDatabases$7(ExternalCatalog.java:91) ~[amoro-ams-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.table.TableMetaStore.call(TableMetaStore.java:268) ~[amoro-common-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.table.TableMetaStore.doAs(TableMetaStore.java:241) ~[amoro-common-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.server.catalog.ExternalCatalog.doAs(ExternalCatalog.java:153) ~[amoro-ams-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.server.catalog.ExternalCatalog.listDatabases(ExternalCatalog.java:89) ~[amoro-ams-0.8.0-incubating.jar:0.8.0-incubating]
amoro-ams     at org.apache.amoro.server.dashboard.controller.TableController.getDatabaseList(TableController.java:595) ~[amoro-ams-0.8.0-incubating.jar:0.8.0-incubating]

Jul 15 '25 15:07 nferrario