WIP - Setup wet run testing

[Dhananjhay]

Summary This PR introduces a fully-automated, label-driven GitHub Actions workflow that:

• Spins up a new ephemeral OpenStack VM on Compute Canada (Arbutus)
• Registers it as a self-hosted GitHub Actions runner
• Waits for the runner to come online
• Downloads wet run test data
• Executes the Hippunfold “wet-run” test inside a Miniforge + Mamba environment
• Tears down the VM and deregisters the runner (success or failure)
• Deletes the runner registration record from GitHub

The workflow is triggered only when the PR is labeled ready-for-wet-run.

To check/do:

• Concurrency amongst different PRs
• Update wet run tests:

1. Side-by-side of the QC results
2. Distribution of vertex-vertex euclidean distances

• Caching Conda environments or pre-baking them into a VM image can significantly speed up repeated runs. However, there are two key bottlenecks:

  Persistent volume mounting: The ideal setup would involve storing cached Conda environments on a persistent volume and mounting it during each run. But due to strict volume limits (e.g., a maximum of two attached volumes per instance), this approach does not scale well when running multiple ephemeral runners in parallel.

  Pre-baked Conda environments in the base VM image: While this avoids external volume limitations, it bloats the base image, resulting in slower VM boot times and increased snapshot size. This adds overhead during runner startup and can even cause timeouts in CI workflows.

• Use GPUs to accelerate the pipeline

[Dhananjhay]

Hey all @akhanf @jordandekraker @mackenziesnyder! I was able to orchestrate an entire end to end wet run testing pipeline and I'm excited to share a working example with you all! There's still work to be done but let me know if anyone has any suggestions or feedback!

[jordandekraker]

Looks awesome, and good to start with 1 lowRes sample for testing. I suppose the next step would be to view a side-by-side of the QC results with the main or target of a given PR. Do you know a good way to do this? In addition to that, we could also check the distribution of vertex-vertex euclidean distances (this could be a new PR though)

[Dhananjhay]

Sounds good. I don't know the answer to that from the top of my head, but I'll add that to the todo list agenda and research on it. Thank you!

[github-actions[bot]]

🔬 Wet run QC result

A side-by-side hippocampal subfield volume plot was generated:

• Left: main reference
• Right: this PR

🖼️ Artifact: sub-01_qc_comparison
📥 Download from "Artifacts" tab in Actions

Let us know if anything looks off!

[Dhananjhay]

This generally looks good to me! The commit history is a bit messy so maybe could use a rebase to clean-up?

Hey @akhanf! Rebasing didn't help clean up the commit history, it's really messy 🥲 so instead I've created a new PR (#521) which was branched off from dev-v2.0.0. It's much easier to see the changes pertaining to wet run ephemeral test logic now. I can close this PR once you approve!