[Spanner] google_default_credentials should probably be async

[pjenvey]

When creating/connecting a Spanner client, there's potentially 2 blocking operations required:

1. Creating/connecting? creds ChannelCredentials::google_default_credentials()
2. Creating/connecting? a grpc Channel/client ChannelBuilder::secure_connect

One of these is likely a potentially blocking operation, probably 1. as grpc's docs show C#'s google_default_credentials() equivalent as async:

using Grpc.Auth;  // from Grpc.Auth NuGet package
...
// Loads Google Application Default Credentials with publicly trusted roots.
var channelCredentials = await GoogleGrpcCredentials.GetApplicationDefaultAsync();

var channel = new Channel("greeter.googleapis.com", channelCredentials);
var client = new Greeter.GreeterClient(channel);
...

The grpcio lib does not provide async versions of these methods. They may potentially be blocking our async SpannerConnection::connect method, which could be problematic.

We should probably contact Google's grpc client team for advise here and followup with grpc-rs project.

[fzzzy]

I was able to make the block alarm work, so we could use it to detect any other places like this in the code. https://github.com/fzzzy/block-alarm

[pjenvey]

I haven't been able to trigger block_alarm on the latest master even with the lack of async here.

Tracing the c# version it goes:

• GoogleGrpcCredentials.GetApplicationDefaultAsync()
  • GoogleCredentials.GetDefaultCredentialAsync() (google-api-dotnet-client lib)
    • loads from a cached credential or calls looks around from different sources

With the caching this probably rarely blocks (maybe only the first time).

Our rust code:

• google_default_credentials()
  • rpc_sys::grpc_google_default_credentials_create() (involving various calls deep into grpc creds code)

The composite credentials code refers to a composite_call_metadata_cb method, suggesting some kind of async callback. Without more digging I'm not sure if there's any caching similar to the C# going on here or any potential blocking code that'd be triggered on every call.

[pjenvey]

Until we figure out how this call might block we'll defer it to actix-web's thread pool and leave this issue open to follow up on it later.