Intermittent I/O timeout connecting from GKE

[blindrood]

trafficstars

Bug Description

Hello, we're using sql proxy as a sidecar, here's a snippet

initContainers:
  - name: cloud-sql-proxy
    restartPolicy: Always
    image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.16.0
    args:
      - "--private-ip"
      - "PROJECT:europe-west1:INSTANCE"

This generally worked fine for us for last 6 months and still works fine. However, recently, when deploying new version of app, new container has problems starting up proxy. Error below. Issue persists for 15-20 min and eventually pod starts up and serves new app fine

Example code (or command)

No response

Stacktrace

Error starting proxy: [PROJECT:europe-west1:INSTANCE] Unable to mount socket: failed to get instance: refresh error: failed to get instance metadata (connection name = "PROJECT:europe-west1:INSTANCE"): Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT/instances/INSTANCE/connectSettings?alt=json&prettyPrint=false": dial tcp: lookup [sqladmin.googleapis.com](http://sqladmin.googleapis.com/): i/o timeout

[PROJECT:europe-west1:INSTANCE] could not resolve instance version: failed to get instance: refresh error: failed to get instance metadata (connection name = "PROJECT:europe-west1:INSTANCE"): Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT/instances/INSTANCE/connectSettings?alt=json&prettyPrint=false": dial tcp: lookup [sqladmin.googleapis.com](http://sqladmin.googleapis.com/): i/o timeout

The proxy has encountered a terminal error: unable to start: [PROJECT:europe-west1:INSTANCE] Unable to mount socket: failed to get instance: refresh error: failed to get instance metadata (connection name = "PROJECT:europe-west1:PROJECT"): Get "https://sqladmin.googleapis.com/sql/v1beta4/projects/PROJECT/instances/INSTANCE/connectSettings?alt=json&prettyPrint=false": dial tcp: lookup [sqladmin.googleapis.com](http://sqladmin.googleapis.com/): i/o timeout

Steps to reproduce?

It sometimes (for now had it 3 times) happens when new version of deployment is released (app image tag changed)

Environment

everything is in one VPC, GKE 1.32.3-gke.1785003 Cloud SQL Proxy version (./cloud-sql-proxy --version): :2.16.0 We had this issue with version 2.12 and recently upgraded to 2.16

Additional Details

No response