s3 type secrets created using credential_chain provider not working for iam role attached to the pod in EKS

[shashideore]

What happens?

running the following test on EKS container with aws iam role attached having permission to s3 fails to access s3

import duckdb
con = duckdb.connect()
con.execute("INSTALL httpfs;")
con.execute("LOAD httpfs;")
con.execute("""
CREATE OR REPLACE SECRET secret (
    TYPE s3,
    PROVIDER credential_chain
);
""")
con.execute("CREATE TABLE test AS SELECT * FROM 's3://my-parquet-file-path/*.parquet'")

error returned duckdb.duckdb.HTTPException: HTTP Error: HTTP GET error on '...' (HTTP 403)

But if we get the credentials from boto3 session and then run the test it works

import duckdb
import boto3
def get_credentials():
    session = boto3.Session()
    credentials = session.get_credentials()
    return {
        "AWS_ACCESS_KEY_ID": credentials.access_key,
        "AWS_SECRET_ACCESS_KEY": credentials.secret_key,
        "AWS_SESSION_TOKEN": credentials.token,  # Optional, may be None for long-term credentials
    }
con = duckdb.connect()
con.execute("INSTALL httpfs;")
con.execute("LOAD httpfs;")
session = boto3.Session()
credentials = session.get_credentials()
credentials = get_credentials()
con.execute(f"SET s3_access_key_id='{credentials['AWS_ACCESS_KEY_ID']}';")
con.execute(f"SET s3_secret_access_key='{credentials['AWS_SECRET_ACCESS_KEY']}';")
con.execute(f"SET s3_session_token='{credentials['AWS_SESSION_TOKEN']}';")
con.execute("CREATE TABLE test AS SELECT * FROM 's3://my-parquet-file-path/*.parquet'")

now it works - but the credentials remain valid for 1 hour only and we like to keep the credentials refreshed

To Reproduce

can you help resolve getting the credentials through credential_chain for iam role attached to pod ?

OS:

GNU/Linux x86_64

DuckDB Version:

1.1.3

DuckDB Client:

3.12.4

Hardware:

No response

Full Name:

Shashikant Deore

Affiliation:

SAS

What is the latest build you tested with? If possible, we recommend testing with the latest nightly build.

I have not tested with any build

Did you include all relevant data sets for reproducing the issue?

No - Other reason (please specify in the issue body)

Did you include all code required to reproduce the issue?

• [x] Yes, I have

Did you include all relevant configuration (e.g., CPU architecture, Python version, Linux distribution) to reproduce the issue?

• [x] Yes, I have

[Tmonster]

Hi @shashideore ,

I assume you are using a profile that assumes another role? I think this is similar to https://github.com/duckdb/duckdb-aws/issues/101 and https://github.com/duckdb/duckdb-aws/issues/89

For now my advice is to change

CREATE OR REPLACE SECRET secret (
    TYPE s3,
    PROVIDER credential_chain
);

to

CREATE SECRET s3table_secret (
    TYPE s3,
    PROVIDER credential_chain,
    CHAIN 'sts',
    ASSUME_ROLE_ARN 'arn:aws:iam::<account_id>:role/<assumed_role>'
);

and see what happens.

we are working on a fix for when you use a profile in /path/to/credentials/file that assumes a role

[shashideore]

Hi @Tmonster , Thanks for looking into the issue. The secret creation works with latest version of duckdb but when try to access s3 con.execute("CREATE TABLE test AS SELECT * FROM 's3://my-parquet-file-path/*.parquet'")

it fails with (HTTP 403 Forbidden) Authentication Failure - this is usually caused by invalid or missing credentials.

• No credentials are provided.
• See https://duckdb.org/docs/stable/extensions/httpfs/s3api.html

[ruiyang2015]

we also facing issue when trying to use the sts role, and i tried to add scope to the secret (when use config as provider, we can set scope), and getting this error:

CREATE SECRET s3table_secret (
    TYPE s3,
    PROVIDER credential_chain,
    CHAIN 'sts',
    ASSUME_ROLE_ARN 'arn:aws:iam::<account_id>:role/<assumed_role>',
    SCOPE 's3://mybcket/myprefix'
);

_duckdb.Error: Secret Validation Failure: during `generate` using the following:
Credential Chain: 'sts'
Role-arn: 'arn:aws:iam::<my account id>:role/myrole'

[dsu4rez]

Same issue here…

[Serpan07]

Same issue here. Do we have any workaround? When is it planned to be fixed?

[dsu4rez]

Same issue here. Do we have any workaround? When is it planned to be fixed?

The ugly workaround is to get "frozen" credentials via boto3 and use them to authenticate via duckdb. Just keep in mind that those credentials are temporary.

[franck-f-b]

Same issue here