Allow authenticating to GCP via service account provided by compute engine

[yuvipanda]

Very cool project! Am excited to play with it :)

I'm trying to set this up with GCP, on a kubernetes cluster with workload identity setup to provide GCP IAM permissions to workloads. All the google cloud sdk clients (including the python one used by this project) know how to use this to 'automatically' find credentials and other metadata (see https://cloud.google.com/docs/authentication/production#automatically). However, because in lithops we are explicitly looking for a service account JSON key file (https://github.com/lithops-cloud/lithops/blob/03374840c07a24f99fe2fb218da1f6f31ccd2abf/lithops/serverless/backends/gcp_cloudrun/cloudrun.py#L48), this automatic behavior of the underlying libraries is overriden, and I get the following error when trying to run the hello world:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/lithops/serverless/backends/gcp_cloudrun/config.py:96, in load_config(config_data)
     94     if param not in config_data['gcp']:
     95         msg = "{} is mandatory under 'gcp' section of the configuration".format(REQ_PARAMS)
---> 96         raise Exception(msg)
     98 if not exists(config_data['gcp']['credentials_path']) or not isfile(config_data['gcp']['credentials_path']):
     99     raise Exception("Path {} must be service account "
    100                     "credential JSON file.".format(config_data['gcp']['credentials_path']))

Exception: ('project_name', 'service_account', 'credentials_path', 'region') is mandatory under 'gcp' section of the configuration

for the following code:

config = {'lithops': {'backend': 'gcp_cloudrun', 'storage': 'gcp_storage'},
          'gcp':  {},
          'gcp_storage': {'storage_bucket': 'leap-scratch',
                      }}

def hello_world(name):
    return 'Hello {}!'.format(name)

if __name__ == '__main__':
    fexec = lithops.FunctionExecutor(config=config)
    fexec.call_async(hello_world, 'World')
    print(fexec.get_result())

    

If the auth params (service account key, etc) are not explicitly set, lithops should try to have the gcp libraries auto discover them. This would work both with application default credentials as well as on compute engine / GKE, and would make running this far simpler in those cases.

Thank you for working on this!

[JosepSampe]

@yuvipanda Good finding! I think PR #964 should be enough. You can test it using master branch

[yuvipanda]

@JosepSampe thank you for the quick response! Unfortunately the error still persists, and is a little different:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/lithops/serverless/backends/gcp_functions/config.py:87, in load_config(config_data)
     85     if param not in config_data['gcp']:
     86         msg = f"{param} is mandatory under 'gcp' section of the configuration"
---> 87         raise Exception(msg)
     89 if 'credentials_path' not in config_data['gcp']:
     90     if 'GOOGLE_APPLICATION_CREDENTIALS' in os.environ:

Exception: project_name is mandatory under 'gcp' section of the configuration

If I add project_name, I get:

File /srv/conda/envs/notebook/lib/python3.9/site-packages/lithops/serverless/backends/gcp_functions/config.py:87, in load_config(config_data)
     85     if param not in config_data['gcp']:
     86         msg = f"{param} is mandatory under 'gcp' section of the configuration"
---> 87         raise Exception(msg)
     89 if 'credentials_path' not in config_data['gcp']:
     90     if 'GOOGLE_APPLICATION_CREDENTIALS' in os.environ:

Exception: service_account is mandatory under 'gcp' section of the configuration

It doesn't need credentials_path, but still needs 'region'. I think at least project name and serviceaccount should be automatically picked too

[JosepSampe]

For now, in this initial step, project_name, service_account and region are still mandatory.

https://github.com/lithops-cloud/lithops/blob/master/docs/source/compute_config/gcp_cloudrun.md#google-cloud-platform

[JosepSampe]

@yuvipanda With #965 project_name and service_account are no longer required. region is still mandatory and I'm not sure if we can remove it: for now I don't see a way to get it from the environment