Automate upgrade testing

[theunrepentantgeek]

trafficstars

For the Beta-0 release, @matthchr manually tested that resources deployed with Alpha-5 would correctly operate with the new release. We should automate these tests so that we can ensure we don't accidentally break backwards compatibility in any future release.

[theunrepentantgeek]

@matthchr summarized his testing as follows:

$ task controller:kind-create
$ task controller:install-cert-manager
$ kubectl apply --server-side=true -f https://github.com/Azure/azure-service-operator/releases/download/v2.0.0-alpha.6/azureserviceoperator_v2.0.0-alpha.6.yaml
$ task controller:make-sp-secret
$ k apply -f v2/config/samples/resources/v1alpha1api20200601_resourcegroup.yaml

// Note, had to change the name in this resource as one already exists
$ k apply -f v2/config/samples/storage/v1alpha1api20210401_storageaccount.yaml

// Confirm the resources were created in Azure portal

// Now upgrade to beta
$ task controller:docker-push-local
$ task controller:install

// Confirm that the deployment is now using the localhost container image

// Confirmed that kubectl get returns the beta version (even though resource was created w/ alpha). This is expected.

// Test creation of beta resource (I modified this sample locally to have a different name so it didn't clash with the one I created above, but it's rooted to the alpha resource group just to make sure that works)
$ k apply -f v2/config/samples/storage/v1beta20210401_storageaccount.yaml

// Confirm edit of alpha resource works and shows up in azure. I edited the edit v2/config/samples/resources/v1alpha1api20210401_storageaccount.yaml to add tags and reapplied

// Confirmed there were no error logs in controller logs

[matthchr]

Or for testing Helm chart upgrade:

$ task controller:kind-create
$ task controller:install-cert-manager

// Add the existing Helm repo to install the latest public chart
$ helm repo add aso2 https://raw.githubusercontent.com/Azure/azure-service-operator/main/v2/charts
$ helm repo update
$ helm upgrade --install --devel aso2 aso2/azure-service-operator \
     --create-namespace \
     --namespace=azureserviceoperator-system \
     --set azureSubscriptionID=$AZURE_SUBSCRIPTION_ID \
     --set azureTenantID=$AZURE_TENANT_ID \
     --set azureClientID=$AZURE_CLIENT_ID \
     --set azureClientSecret=$AZURE_CLIENT_SECRET

$ k apply -f v2/config/samples/resources/v1beta20200601_resourcegroup.yaml
// Note, had to change the name in this resource as one already exists
$ k apply -f v2/config/samples/storage/v1beta20210401_storageaccount.yaml

// Confirm the resources were created in Azure portal

// Generate a fresh chart locally.
$ task controller:gen-helm-manifest

// Push a local docker image
$ task controller:docker-push-local

// Install the local Helm chart with the local docker image
$ helm upgrade --devel aso2 ./v2/charts/azure-service-operator \
     --namespace=azureserviceoperator-system \
     --set image.repository=localhost:5000/azureserviceoperator:latest \
     --set azureSubscriptionID=$AZURE_SUBSCRIPTION_ID \
     --set azureTenantID=$AZURE_TENANT_ID \
     --set azureClientID=$AZURE_CLIENT_ID \
     --set azureClientSecret=$AZURE_CLIENT_SECRET

// Confirm that the deployment is now using the localhost container image

// Test creation of beta resource (I modified this sample locally to have a different name so it didn't clash with the one I created above, but it's rooted to the alpha resource group just to make sure that works)
$ k apply -f v2/config/samples/storage/v1beta20210401_storageaccount.yaml

// Confirmed there were no error logs in controller logs