calrissian
calrissian copied to clipboard
Run conformance tests in CI
The conformance tests should run automatically under a CI platform, like the other implementations listed on https://ci.commonwl.org.
Calrissian uses CircleCI for running automated tests. Through much trial and error I was able to get a minikube cluster running in CircleCI (via https://circleci.com/orbs/registry/orb/circleci/kubernetes#usage-deployment) and attempt to run the tests/collect artifacts. See branch try-circle-minikube.
Two issues are blocking progress:
- The CircleCI VM only has 2 CPUs, leaving <2CPUs for kubernetes pods. Some CWL conformance tests (including N1) require 2CPUs, so these pods get stuck waiting. I couldn't increase this limit or force it to a higher value
- Extracting the results/artifacts requires some additional resource creation/cleanup. Currently the junit XML file lives inside the cluster, so it's not available to the CircleCI environment to upload it. This is just a scripting hurdle.
I'm going to park this here for now and may explore running these tests in another mode.