clusterfuzz icon indicating copy to clipboard operation
clusterfuzz copied to clipboard

[Deployment revisited][Staging][PoC] Fix create config

Open vitorguidi opened this issue 9 months ago • 0 comments

Create config has gotten stale over time. A few encountered issues:

  • Redis is being created both by terraform and deployment manager, which leads to butler deploy failing after the bootstrap because the resource already exists. Now it will only be created by terraform
  • [TODO] Now that redis is terraform responsibility, it should also create the serverless connector in the same VPC. This was done in _update_redis
  • Redis must be created BEFORE deploying appengine, because there is a dependency on the REDIS_HOST env var. deploy_terraform and deploy_k8s were moved to before deploy_appengine
  • The command line browser used in the CLI fails to authenticate in the oauth flow due to javascript not being available. use_browser was set to false, so the operator manually accesses the provided link
  • The K8S_PROJECT env var is not bootstrapped in create config, so deploy_k8s fails
  • WindowRateLimitTask index is absent in src/appengine.yaml, so fuzz task fails
  • Secrets api is disabled, and the GCS signer secret is absent (manually created) > need to create a service account and paste the json token as a secret
  • Batch pools are reference but not created, which breaks schedule_utask_main (and possibly schedule_fuzz_task)
  • The redis serverless connector is created in the default VPC, but appengine/redis themselves are on the main one, which makes redis accesses timeout
  • Batch definitions are out of date with schedule-fuzz and schedule-utask-main: fields in batch.yaml differ between createa config and the script that uses them
  • Preprocess, utask_man and postprocess queues were missing in the bootstrapping config

This PR addresses the above issues

Part of #4728

vitorguidi avatar Mar 14 '25 13:03 vitorguidi