provider-gcp icon indicating copy to clipboard operation
provider-gcp copied to clipboard

CloudSQLInstance contantly being updated

Open bartse opened this issue 2 years ago • 7 comments

What happened?

  1. Using crossplane 1.9.0 + gcp provider 0.21.0 running on GKE 1.21
  2. Setup a CloudSQLInstance using below config
...
  forProvider:
    databaseVersion: POSTGRES_14
    instanceType: CLOUD_SQL_INSTANCE
    maxDiskSize: 40
    region: europe-west1
    settings:
      activationPolicy: ALWAYS
      availabilityType: ZONAL
      backupConfiguration:
        startTime: "04:00"
      dataDiskSizeGb: 10
      dataDiskType: PD_SSD
      ipConfiguration:
        privateNetwork: projects/<project-id>/global/networks/<network-name>
        requireSsl: true
      maintenanceWindow:
        day: 6
        hour: 23
        updateTrack: stable
      pricingPlan: PER_USE
      replicationType: SYNCHRONOUS
      storageAutoResize: true
      tier: db-custom-2-3840
...
  1. The provider is constantly updating the instance to a point where it fails
  Type     Reason                        Age                   From                                                 Message
  ----     ------                        ----                  ----                                                 -------
  Normal   CreatedExternalResource       5m46s                 managed/cloudsqlinstance.database.gcp.crossplane.io  Successfully requested creation of external resource
  Warning  CannotUpdateExternalResource  81s (x12 over 103s)   managed/cloudsqlinstance.database.gcp.crossplane.io  cannot update the CloudSQL instance: googleapi: Error 409: Operation failed because another opera
tion was already in progress. Try your request after the current operation is complete., operationInProgress
  Normal   UpdatedExternalResource       42s (x18 over 5m46s)  managed/cloudsqlinstance.database.gcp.crossplane.io  Successfully requested update of external resource
image

I did find earlier bug reports but those seem to have been fixed a while ago so not sure anyone else is experiencing this issue.

  • https://github.com/crossplane-contrib/provider-gcp/issues/121
  • https://github.com/crossplane-contrib/provider-gcp/issues/352
  • PR fix https://github.com/crossplane-contrib/provider-gcp/pull/374

bartse avatar Aug 20 '22 08:08 bartse

I am seeing the same behavior using the same versions of crossplane, gcp-provider en GKE as @bartse. I had to disable public network ( for which i couldn't find an option in crossplane). Afterwards the DB is constantly being updated.

miglocatiq avatar Sep 01 '22 09:09 miglocatiq

Happens for us, too. Constant update leads to 400 error messages per 5 minutes and makes it impossible to actually maintain the database in a stable way.

pschmidbauer avatar Nov 10 '22 15:11 pschmidbauer

I've encountered the same issue when working with this. I'm using the full URI for the private network, so it can't be that. I've tried to look through GCP logs to find out what the differences are between the two instance versions (the existing one and the one on which crossplane detects a change), and the only things that I found are these arguments:

connection_mode: ALLOW_ALL - is going to be added ssl_params { <... omitted ...>   } - getting removed I've tried to look in the documentation to see if these things can be specified, but I had no success.

CalinFlorescu avatar Nov 10 '22 16:11 CalinFlorescu

The same issue is on the same provider version. We thought it was related failover replica, but looks like no. Any idea?

insider89 avatar Nov 18 '22 14:11 insider89

I am experiencing the same. Deletion of CloudSQLInstance object and recreation of it solve the issue, but the initial behavior happens to us randomly

CarpathianUA avatar Nov 18 '22 14:11 CarpathianUA