[Bug]: How to configure S3 bucket access for Paimon catalog via IAM Role ARN (WebIdentity)? Is it supported?

[Sbaia]

What happened?

Hi, I'm trying to configure a Paimon catalog in Amoro with S3 storage, using authentication via IAM Role ARN and Kubernetes Service Account (WebIdentity). My setup works with explicit AK/SK credentials, but DOES NOT work when switching to role-based authentication.

Here is the configuration I've used:

  "name": "Paimon",
  "type": "hadoop",
  "authConfig": {
    "auth.type": "custom"
  },
  "storageConfig": {
    "storage.type": "S3",
    "fs.s3a.region": "eu-west-1",
    "fs.s3a.aws.credentials.provider": "com.amazonaws.auth.WebIdentityTokenCredentialsProvider"
  },
  "properties": {
    "warehouse": "s3a://my-bucket-in-aws/"
  },
  "tableProperties": {},
  "tableFormatList": ["PAIMON"]
}```

- With AK/SK credentials, it works fine.
- Using ARN/ServiceAccount, setting fs.s3a.aws.credentials.provider to com.amazonaws.auth.WebIdentityTokenCredentialsProvider, I always get access error or "Anonymous" authentication.
- The S3A scheme is required for Hadoop integration, otherwise I get a missing filesystem implementation error. (Not sure if that's related to the ARN issue.)
- The IAM role is attached and correctly configured; Flink can write to S3 with the same role and setup.

### Affects Versions

master/0.8.1

### What table formats are you seeing the problem on?

Paimon

### What engines are you seeing the problem on?

AMS

### How to reproduce

try to configure a paimon catalog using AWS Role ARN

### Relevant log output

```shell
2025-11-07 16:06:23,832 ERROR [JettyServerThreadPool-61] [org.apache.amoro.server.dashboard.DashboardServer] [] - An error occurred while processing the url:http://amoro.tools.com/api/ams/v1/catalogs                                      │
│ java.io.UncheckedIOException: java.nio.file.AccessDeniedException: s3a://my-bucket-in-aws/user.sys: getFileStatus on s3a://my-bucket-in-aws/user.sys: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3 │
│ , Status Code: 403, Request ID: JVF0V003PHTCA7R6, Extended Request ID: gjOri4E7uwBNMnoamvlGasM5rzYFBzGlBC7kQhRHVZaoXtGPbG3Yybtq1oy9teKAcOEE2783VpRUDYDzNIwnR/iei6w2wzRn):null                                                                                   │
│     at org.apache.paimon.privilege.FileBasedPrivilegeManager.getTable(FileBasedPrivilegeManager.java:371) ~[paimon-bundle-1.2.0.jar:1.2.0]                                                                                                                      │
│     at org.apache.paimon.privilege.FileBasedPrivilegeManager.getUserTable(FileBasedPrivilegeManager.java:352) ~[paimon-bundle-1.2.0.jar:1.2.0]                                                                                                                  │
│     at org.apache.paimon.privilege.FileBasedPrivilegeManager.privilegeEnabled(FileBasedPrivilegeManager.java:118) ~[paimon-bundle-1.2.0.jar:1.2.0]                                                                                                              │
│     at org.apache.paimon.privilege.PrivilegedCatalog.tryToCreate(PrivilegedCatalog.java:72) ~[paimon-bundle-1.2.0.jar:1.2.0]                                                                                                                                    │
│     at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:74) ~[paimon-bundle-1.2.0.jar:1.2.0]                                                                                                                                          │
│     at org.apache.paimon.catalog.CatalogFactory.createCatalog(CatalogFactory.java:67) ~[paimon-bundle-1.2.0.jar:1.2.0]                                                                                                                                          │
│     at org.apache.amoro.formats.paimon.PaimonCatalogFactory.paimonCatalog(PaimonCatalogFactory.java:87) ~[amoro-format-paimon-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]                                                                                                    │
│     at org.apache.amoro.formats.paimon.PaimonCatalogFactory.create(PaimonCatalogFactory.java:79) ~[amoro-format-paimon-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]                                                                                                           │
│     at org.apache.amoro.formats.paimon.PaimonCatalogFactory.create(PaimonCatalogFactory.java:42) ~[amoro-format-paimon-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]                                                                                                           │
│     at org.apache.amoro.CommonUnifiedCatalog.initializeFormatCatalogs(CommonUnifiedCatalog.java:212) ~[amoro-common-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]                                                                                                              │
│     at org.apache.amoro.CommonUnifiedCatalog.<init>(CommonUnifiedCatalog.java:60) ~[amoro-common-0.9-SNAPSHOT.jar:0.9-SNAPSHOT]                                                                                                                                 │
│     at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                  │
│     at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:52) ~[javalin-4.6.8.jar:4.6.8]                                                                                                                                             │
│     at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                   │
│     at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                         │
│     at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                  │
│     at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                   │
│     at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                  │
│     at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                      │
│     at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                              │
│     at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                    │
│     at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                                            │
│     at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                         │
│     at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                                 │
│     at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                                   │
│     at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                         │
│     at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                             │
│     at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                                       │
│     at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                                    │
│     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                │
│     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                              │
│     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                             │
│     at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                    │
│     at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                              │
│     at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                      │
│     at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.51.v20230217.jar:9.4.51.v20230217]                                                                                                                 │
│     at java.lang.Thread.run(Thread.java:829) [?:?]                                                                                                                                                                                                              │
│ Caused by: java.nio.file.AccessDeniedException: s3a://my-bucket-in-aws/user.sys: getFileStatus on s3a://my-bucket-in-aws/user.sys: software.amazon.awssdk.services.s3.model.S3Exception: null (Service: S3, Status Code: 403, │
│  Request ID: JVF0V003PHTCA7R6, Extended Request ID: gjOri4E7uwBNMnoamvlGasM5rzYFBzGlBC7kQhRHVZaoXtGPbG3Yybtq1oy9teKAcOEE2783VpRUDYDzNIwnR/iei6w2wzRn):null                                                                                                      │
│     at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:266) ~[hadoop-aws-3.4.0.jar:?]                                                                                                                                                        │
│     at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:155) ~[hadoop-aws-3.4.0.jar:?]

Anything else

Always

Are you willing to submit a PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [x] I agree to follow this project's Code of Conduct

[netlify[bot]]

✅ Deploy Preview for ethereumclassic ready!

Name	Link
🔨 Latest commit	a6639f9c3c4d0821b2893bbec193f2a6dd090279
🔍 Latest deploy log	https://app.netlify.com/projects/ethereumclassic/deploys/6920c0bcf343010009ed00b8
😎 Deploy Preview	https://deploy-preview-1651--ethereumclassic.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.