Azure upstream

[violetbrina]

Bug fixes to enable Azure deployment

Most of these bugs were discovered in deploying the MySQL server from scratch, specifically deploying version 8.0. Some were encountered when we hit certificate issues in trying to run the ./bootstrap.sh deploy_unmanaged step multiple times within 24hrs. Documentation was clarified in order to resolve this issue.

• build.yaml

  • Step one fails on rerun since the /repo directory exists, -p to fix

• ci/create_database.py

  • In MySQL 8 a new error was introduced 4006
  • This error gets triggered on the CREATE USER IF NOT EXISTS commands for both user and admin if the user was previously created and set a a definer on any events/triggers.
  • Really this statement should be a no-op given that the user exists, but for some reason the error triggers anyway.
  • To get around this I added a manual check if the user/admin exists and if they do simply skip the create user command. This fixes the bug and allows the MySQL db deploy to finish properly

• dev-docs/letsencrypt.md

  • Debugging was confusing since the revoke command addressed ids we were unable to find.
  • After extensive searching I added to the documentation how to find your existing cert IDs if you need to revoke them.

• infra/azure/README.md

  • Added clarity to the Azure deployment documentation

• infra/azure/bootstrap.sh

  • Added the passing of additional flag arguments to terraform
  • In our case the passing of the -upgrade flag to the terraform init step was required in order to continue

• infra/azure/main.tf, infra/azure/modules/batch/main.tf, infra/azure/modules/batch/variables.tf infra/azure/variables.tf

  • Add additional argument for the az_storage_account.
  • The name must be globally unique in Azure, so the original argument failed on our deployment since it shared the name with the Hail team's Azure deployment

[violetbrina]

Apologies for the delay in addressing the feedback. Had some urgent projects to attend to. Here are the changes you've suggested. Let me know if there's anything else I can do to help with this PR.

[danking]

@daniel-goldstein back to you

[daniel-goldstein]

@violetbrina there are just a couple of lint errors left on this branch. If you locally run make check-ci you should see the couple of suggestions to fix. I'm also happy to push the fixes to populationgenomics:azure-upstream if that's fine.

[danking]

Closing in favor of https://github.com/hail-is/hail/pull/12521 which has the lints fixed.