lance icon indicating copy to clipboard operation
lance copied to clipboard

Fix the behavior of auto-loading credentials chain on S3

Open eddyxu opened this issue 1 year ago • 2 comments

Problem Statement

Currently, Lance does not support a "default chain of credentials lookups", which makes the experience on AWS bad.

AWS rust sdk has the desired functionalty https://docs.rs/aws-config/latest/aws_config/default_provider/credentials/struct.DefaultCredentialsChain.html

Not sure how to integrate into object_store rust crate yet.

eddyxu avatar May 29 '23 17:05 eddyxu

There was a thread in Arrow-rs (i forgot where) that talks about how pulling in the AWS SDK makes the crate really big / complicated. I don't remember whether it was on github or discord. But it's something to watch out for.

changhiskhan avatar May 29 '23 17:05 changhiskhan

quick note: object_store only recently exposed the configurbility of credentials field https://github.com/apache/arrow-rs/pull/4235

Going to need to upgrade arrow-rs for this one

chebbyChefNEQ avatar May 30 '23 14:05 chebbyChefNEQ