"failed to connect to instance: failed to get instance: Refresh error: failed to get instance metadata..." intermittent connection errors

[aronsoderling]

trafficstars

Bug Description

Running cloud-sql-proxy as a sidecar container in GKE with CSQL_PROXY_AUTO_IAM_AUTHN=true and CSQL_PROXY_CREDENTIALS_FILE=PATH_TO_CREDENTIALS intermittently fails to connect to the database instance with the following error message:

failed to connect to instance: failed to get instance: Refresh error: failed to get instance metadata (connection name = "INSTANCE_CONNECTION_NAME"): googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

With ACCESS_TOKEN_EXPIRED as reason in the error details (see stacktrace).

Subsequent connection attempts succeed.

This doesn't happen very often, we're seeing roughly 1 error per 10000 requests in our system over a month. Can't really say how many connection attempts those requests corresponds to. We've been seeing these errors randomly for quite some time, at least for the past three months.

Example code (or command)

Configuration looks like this:

CSQL_PROXY_PORT=5432
CSQL_PROXY_AUTO_IAM_AUTHN=true
CSQL_PROXY_CREDENTIALS_FILE=PATH_TO_CREDENTIALS
CSQL_PROXY_QUIET=true
CSQL_PROXY_HEALTH_CHECK=true
CSQL_PROXY_HTTP_PORT=9801
CSQL_PROXY_HTTP_ADDRESS=0.0.0.0
CSQL_PROXY_EXIT_ZERO_ON_SIGTERM=true
CSQL_PROXY_STRUCTURED_LOGS=true`

Stacktrace

failed to connect to instance: failed to get instance: Refresh error: failed to get instance metadata (connection name = "INSTANCE_CONNECTION_NAME"): googleapi: Error 401: Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.

Details:
[
  {
    "@type": "type.googleapis.com/google.rpc.ErrorInfo",
    "domain": "googleapis.com",
    "metadata": {
      "method": "google.cloud.sql.v1beta4.SqlConnectService.GetConnectSettings",
      "service": "sqladmin.googleapis.com"
    },
    "reason": "ACCESS_TOKEN_EXPIRED"
  }
]
{}
More details:
Reason: authError, Message: Invalid Credentials

Steps to reproduce?

I haven't been able to reproduce this locally.

Environment

1. Docker
2. gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.13.0
3. cloud-sql-proxy INSTANCE_CONNECTION_NAME

Additional Details

I found this issue https://github.com/GoogleCloudPlatform/cloud-sql-proxy/issues/2212 which sounds similar.