calrissian icon indicating copy to clipboard operation
calrissian copied to clipboard

Run conformance tests in CI

Open dleehr opened this issue 5 years ago • 0 comments

The conformance tests should run automatically under a CI platform, like the other implementations listed on https://ci.commonwl.org.

Calrissian uses CircleCI for running automated tests. Through much trial and error I was able to get a minikube cluster running in CircleCI (via https://circleci.com/orbs/registry/orb/circleci/kubernetes#usage-deployment) and attempt to run the tests/collect artifacts. See branch try-circle-minikube.

Two issues are blocking progress:

  1. The CircleCI VM only has 2 CPUs, leaving <2CPUs for kubernetes pods. Some CWL conformance tests (including N1) require 2CPUs, so these pods get stuck waiting. I couldn't increase this limit or force it to a higher value
  2. Extracting the results/artifacts requires some additional resource creation/cleanup. Currently the junit XML file lives inside the cluster, so it's not available to the CircleCI environment to upload it. This is just a scripting hurdle.

I'm going to park this here for now and may explore running these tests in another mode.

dleehr avatar Jul 24 '19 18:07 dleehr