Using S3 creds in storage_options results in side effect of setting env vars

[alexr-bq]

Environment

Delta-rs version: 0.14.0

Binding: Rust

Environment: macOS Ventura 13.4

---

Bug

What happened:

When passing in S3 creds via storage options to create a delta table, ie AWS_ACCESS_KEY_ID, this has an unintended side-effect of overriding set environment variables.

What you expected to happen:

S3 values to be used from the storage options map without setting any environment variables.

How to reproduce it:

let storage_options_invalid: HashMap<String, String> = HashMap::from([ ( "AWS_DEFAULT_REGION".to_string(), env::var("AWS_DEFAULT_REGION").unwrap().to_string(), ), ( "AWS_SECRET_ACCESS_KEY".to_string(), env::var("AWS_SECRET_ACCESS_KEY").unwrap().to_string(), ), ("AWS_ACCESS_KEY_ID".to_string(), "INVALID123".to_string()), ( "AWS_SESSION_TOKEN".to_string(), env::var("AWS_SESSION_TOKEN") .unwrap_or("".to_string()) .to_string(), ), ]);

println!("{}",env::var("AWS_ACCESS_KEY_ID").unwrap().to_string()); // Prints correct value

let mut table = deltalake::open_table_with_storage_options( table_filepath.as_str(), storage_options_invalid, )

println!("{}",env::var("AWS_ACCESS_KEY_ID").unwrap().to_string()); // Prints "INVALID123"

[r3stl355]

It happens here and looks like there is a good reason according to comments https://github.com/delta-io/delta-rs/blob/bdf1c4e765ca457e49d4fa53335d42736220f57f/rust/src/storage/s3.rs#L259. It should only effect the current execution context - not the global environment (e.g. on running the again the correct values will be picked up from global env again).

It also looks like there is a bit of a delay before the request actually fails (some kind of caching?). I mean running a test with right value of AWS_ACCESS_KEY_ID set in the storage option followed by several runs with invalid value still works.

Actually, other values (AWS_SECRET_ACCESS_KEY and AWS_SESSION_TOKEN) show the same behavoiur