lance
lance copied to clipboard
Fix the behavior of auto-loading credentials chain on S3
Problem Statement
Currently, Lance does not support a "default chain of credentials lookups", which makes the experience on AWS bad.
AWS rust sdk has the desired functionalty https://docs.rs/aws-config/latest/aws_config/default_provider/credentials/struct.DefaultCredentialsChain.html
Not sure how to integrate into object_store
rust crate yet.
There was a thread in Arrow-rs (i forgot where) that talks about how pulling in the AWS SDK makes the crate really big / complicated. I don't remember whether it was on github or discord. But it's something to watch out for.
quick note: object_store
only recently exposed the configurbility of credentials
field https://github.com/apache/arrow-rs/pull/4235
Going to need to upgrade arrow-rs
for this one