Improve how Cloud Storage presigned requests are generated

[atrauzzi]

Currently, pre-signing requests for cloud storage is - for lack of better words - a bit painful.

I would love to see it as simple as:

var credential = (await GoogleCredential.GetDefaultServiceAccount()) ?? throw new("No service account assigned.");
var urlSigner = await UrlSigner.FromServiceAccountCredential(credential);
var uri = await urlSigner.SignAsync(someBucket, key, duration, method);

The key focus here is that I think the APIs could afford to be a little more discoverable and self-evident in terms of how a user can obtain whatever credential type that satisfies their situation. The most crucial aspect being that right now, service accounts feel kind of second-class, which inhibits the convenience of being able to assign them to things like Google Cloud Run revisions . :slightly_smiling_face:

[jskeet]

The code you've linked to is for signing without a service account.

If you've got a service account, the code is very nearly what you showed:

UrlSigner urlSigner = UrlSigner.FromServiceAccountCredential(credential);
string url = urlSigner.Sign(bucketName, objectName, TimeSpan.FromHours(1));

So are you really asking for an easier way to obtain a service account credential? It looks like the UrlSigner part is already as you want it...

We can look at that, but I don't think it makes sense to link this specifically to URL signing other than as one example of where service accounts are used.

[atrauzzi]

Indeed! That is for sure a more accurate definition. 🙏

[jskeet]

Okay, so now we know that you're looking for an easier way to obtain a service account, the question is what that should look like. Auth is pretty complex, partly due to the large number of places developer code can run.

One common approach to getting a service account would be something like:

GoogleCredential defaultCredential = GoogleCredential.GetApplicationDefault();
ServiceCredential service = (ServiceCredential) defaultCredential.UnderlyingCredential;

You can also use GoogleCredential.FromFile, or GoogleCredential.FromJson for example - and again, cast the underlying credential.

Please could you give more details about the context in which you'd want to make this simpler?

[atrauzzi]

Ideally, I would like a way to do so without having to use FromFile or FromJson.

My example above does include a bit that might be able to answer your question, something like this perhaps:

var credential = (await GoogleCredential.GetDefaultServiceAccount());

This would be used while running as a Cloud Run application and would - as the name says - get the default service account associated with the Cloud Run revision the code is running as.

[jskeet]

@amanda-tarafa can probably talk to this more, but if you're running on Cloud Run (etc) you'd end up in the .NET client libraries with a ComputeCredential by default - which doesn't have as much information as a ServiceAccountCredential; in particular, it doesn't have the private key which would be used to sign URIs. In some places we make the address of the project-default service account available (or "the service account that is used by a particular API for some purposes", e.g. building containers) but we don't make the full service account details available.

[amanda-tarafa]

AS @jskeet said when running on Cloud Run, actually when running on most Google Cloud environments, the Application Default Credentials is a ComputeCredential which basically means that we use the metadata server to obtain access and id tokens instead of the service account key. This is done precisely to avoid having to share a service account key.

For signing (anything) you do need the service account private key which is what we encapsulate in a ServiceAccountCredential. And right now, the easiest way to get a ServiceAccountCredential when running on Cloud Run is that you download the service account key and inject it in the container and either

a) set the GOOGLE_APPLICATION_CREDENTIALS env variable to the path of the service account key file in which case you can do

GoogleCredential defaultCredential = GoogleCredential.GetApplicationDefault();
ServiceCredential service = (ServiceCredential) defaultCredential.UnderlyingCredential;

or b) you can use UrlSigner.FromServiceAccountPath("path to the service account key").

I know neither is ideal, but it's the easieast workaround I can offer.

If you don't want to download the service account key I might be able to come up with some suggestions, but that will requiere some coding on your part.

[atrauzzi]

Please forgive me, but I think the scope of this conversation keeps getting mixed up. I'm not asking for a compute credential. This ticket isn't about a workaround and what already is, this is about improving the google libraries to make something that doesn't require boilerplate/workaround in user code and making it a little easier to discover via the SDK.

I'm having a hard time understanding why I would need to work with GOOGLE_APPLICATION_CREDENTIALS or key matter for this. The solution found here even manages to do so without me having to populate any extra information. It's just using the metadata server to get a "default service account".

Why not just add a method to the library that obtains the default service account from the metadata server, exactly like the workaround I linked to, but that can return a ServiceAccount credential?

Either way, that's the ask here. I've already implemented the workaround, it's just... Obviously extra code which I've stashed in a conventions libary for my organization, but now has to be maintained and explained to people.

It's a total quality of life thing here.

[amanda-tarafa]

When I said:

If you don't want to download the service account key I might be able to come up with some suggestions, but that will requiere some coding on your part.

I was going to suggest you did exactly what is shown on that code you linked. I hadn't understood you already implemented that, sorry for the confusion.

Why not just add a method to the library that obtains the default service account from the metadata server, exactly like the workaround I linked to, but that can return a ServiceAccount credential?

Because the metadata server does not return a service account credential. In the sample you linked and that you implemented what's happening is (explaining just in case you are curious and to drill down on the point that what you are doing is not signing with the ServiceAccountCredential):

• You obtain the service account id from the metadata server. This is the service account email address, not the service account key. The service account key is what you need to create a ServiceAccountCredential and what you need to sign anything with it.
• You then obtain the default application credentials which in Cloud Run will be ComputeCredentials.
• You then create an IamService that will use the ComputeCredential to call the IAM API.
• You then create a signer (IamServiceBlobSigner) that will use the IAM API, through the IAMService authenticated with the ComputeCredential to sign data in the name of the service account you have the id (email) of. You need to call the API because you don't have the service account private key. And since the ComputeCredential "happens" to represent the same service account as the email you obtained from the metadata server, this is possible.
• Then you use the signer.

We could wrap all of this in a method, and we'll discuss internally whether we want to do something like that and where. We'll come back here with what we decide.

[amanda-tarafa]

Just to make it clearer, your request, which we will consider, is to make it easier to sign without a service account key i.e. without a ServiceAccountCredential as you cannot easily obtain one when running on Cloud Run.

[atrauzzi]

For sure. ♥️

And of course, I'd also love to see things evolve to making it easier to consume the service account authority. Either by exposing a method to obtain the current default service account, or for existing SDK methods that require service accounts to be relaxed into accepting compute credentials, under the assumption that the compute credential could have a service account associated with it.

Part of what is confusing here might be the fact that a service account credential type exists to contrast against compute credentials. People assign service accounts to compute resources exactly to avoid having to back channel additional credentials during deployment.

[amanda-tarafa]

I think you are mixing terms here, mind you, it is confusing, but the distinction between ComputeCredential and ServiceAccountCredential is very intentional:

• You have service accounts, which you assign roles/permissions to and can set up as default application accounts on GCP environments (GCE, Cloud Run etc...)
• You have service account credentials (the .NET class ServiceAccountCredential) which contain a private key (the content of the service account key JSON file) for the service account they represent. Having a private key means that service account credentials are able to sign data, including JWTs to identify themselves. They can do all of this locally without the need to call any API, and the represented service account doesn't need any special signing permissions. The drawback is that you need to securely manage a file that contains your service account private key, and this of course is a security risk.
• You also have compute credentials (the .NET class ComputeCredential) which represents that account (usually a service account) that you configure your GCP environment to run with. The main reason for using compute credentials is that you don't have to share a file containing the private key of your service account. But that means that the compute credential cannot, on its own, sign data or identify itself as the underlying service account. The compute credential relies on GCP metadata servers and/or the IAM API to perform most operations that the service account credential can do on its own. In particular, for signing, the service account needs to have appropiate IAM API permissions and the compute credetial appropiate scopes. When you request we make ComputeCredential interchangeable with ServiceAccountCredential generally, you are requesting that we hide all of this complexity (extra dependencies, permissions, etc. and also security implications) from the user, and this is a very valid request, but it is also one that we won't take lightly as it gives the impression of "things will work out of the box", when they won't.

For now, we are going to try a UrlSigner specific implementation that will take a compute credential.

[atrauzzi]

:confetti_ball: