drone-gae icon indicating copy to clipboard operation
drone-gae copied to clipboard

MaxVersions can error when trying to delete a version with an operation in progress

Open martha opened this issue 3 years ago • 0 comments

Steps to reproduce:

  1. Deploy a GAE backend service with MaxVersions set to 1
  2. Deploy the service again (maybe several times; the bug does not happen consistently). You will see an error from GCP that looks like this:
ERROR: (gcloud.app.versions.delete) Issue deleting version: [service-name/version-num]
--
[service-name/version-num]: ABORTED: Cannot operate on 
apps/project-name/services/service-name/versions/version-num 
because an operation is already in progress for 
apps/project-name/services/service-name/versions/version-num by {hash}.
error: exit status 1

After reaching out to GCP support to ask about this error, we received the following note from them:

If you are running the "remove versions" tool immediately after the version is deployed, we may still be performing operations internally (e.g. stopping the old version, performing traffic splitting for the new version, etc.). In cases where operations are still being performed on the old version, we will reject any other operations (in this case, your delete request). Whether other operations are occurring may also depend on how exactly you are using your CI/CD tool. I would recommend you to delete an old version more than 10 minutes after the new version is deployed and let me know if you still face the issue. We can do further digging in such a case.

Rather than waiting 10 minutes (which is a loong time lag to introduce to our Drone pipelines), this could be fixed by checking if an operation is already in progress and if so, waiting (or skipping) before starting a new operation on a version.

martha avatar Jul 16 '20 19:07 martha