Better authentication for GCP

[djouallah]

currently to connect to gcp, delta table require an environment variable something like this

import os
os.environ["SERVICE_ACCOUNT"] ='secret_API.json'

which is fine when using a notebook for example, but when I tried to deploy it to Google cloud function, it does not works and complain about authentification, error, currently I am just using arrow dataset as it works out of the box , but would love that delta has the same behavior

[sa-]

Allowing access to cloud storage via fsspec should fix most connection/auth issues but I have no idea how much effort it would be to implement this

[renzepost]

Arrow by default follows Application Default Credentials to authenticate the user. It would be great if deltalake supports this as well! Using service account keys is no longer the preferred method for authentication in GCP. We should be able to use workload identity federation.

[houqp]

This is something that we need to add to https://github.com/apache/arrow-rs/tree/master/object_store first. PRs welcome!