MLOpsPython
MLOpsPython copied to clipboard
Discuss: Extend pr.yaml to test everything
This is an issue to discuss an approach we just implemented, and whether I should make a PR here to share it with you guys. It is a bit of work to extract it from our generated clone, so I will only do it if there is actually an interest.
Problem: The pr.yaml today only does superficial testing, and we found ourself comiting a lot of stuff to master which failed at some point in the pipeline (there are many moving parts even with such a small project as this). One of the claimed advatages of "the cloud" and infrastructure as code is that you should be able to dynamically create and drop environments as we need them, so why not get a new environment for your pull requests? So that is what we implemented. To be a bit more precise:
- All the steps in
.pipelines/diabetes_regression-ci.yml
is extracted to a step-template (calledci-steps
). -
pr.yaml
reads the basename from the variable group, but sets all other variables (like resource group name etc) to things like$Base_namePR$PR_numberRG
, e.g.diabetesPR24RG
. -
pr.yaml
starts by usingAzureResourceGroupDeployment
to deploy the full environment, just like in environment_setup/iac-create-environment-pipeline-arm.yml, before it callsci-steps
. - If nothing fails then
pr.yaml
ends by deleting the new resource group (and everything in it). If something fails the resource group remains so you can read logs and stuff.
There is one major implementation "detail", and that is that I am not able to create a workspace service connection automatically, so I had to use the subscription service connection ($AZURE_RM_SVC_CONNECTION
) instead. This worked most places, except to the three calls to the ML tasks installed into azure devops by the azure machine learning extension, those starting with ms-air-aiagility...
(there were no way of telling them what resource group they should work in). So I had to replace them with calls to az ml ...
(related: #291). But with that in place it works, and every PR gets its own environment which runs and tests the full pipeline.
I suspect that this is not particularly safe if you accept PRs from anyone, but I guess its also possible to have github/pipelines only run it if the PR was made by e.g. an admin.
Anyway, is there any interest in this?