Workspace volume that can be persisted between tasks and builds

[fkorotkov]

There are two use cases are not yet ideally solved in Cirrus CI:

1. Cloning of a huge repository
2. Caching dozens of Gb of caches

I wonder if the following feature might solve both of them. Hear me out!

Disclaimer: this will only work for Linux and won't be generic.

What if we allow to created volumes that will be persisted either for a duration of a build or for some TTL after the last access. Under the hood it will be a simple GCP Persistent Disk. Such disks can be mounted in read-only mode to multiple VMs and/or containers which is ideal for PRs which usually generate the most builds and don't need to update either caches or Git snapshots.

For use case #1 it will be possible to mount a disk with some snapshot of the repository and use --reference flag for git clone command to refer it. There can be either a Cron build of the default branch build can update such volume with the reference.

For use case #2 it will be possible to just mount the volume in read-only for PRs and update the caches on default branch builds.

skip: $CIRRUS_CRON != ''

task:
  name: Update Reference
  only_if: $CIRRUS_CRON != ''
  container:
    image: alpine:latest
    volumes:
      - name: clone-reference
        mount: /tmp/clone-reference
        mode: rw
  env:
    CIRRUS_WORKING_DIR: /tmp/clone-reference
  clone_script: git pull

# global env that will be supported by the agent to configure the reference for the default clone instrunction
env:
  CIRRUS_CLONE_REFERENCE: /tmp/clone-reference

task:
  name: Clone Monorepo
  container:
    image: alpine:latest
    volumes:
      - name: clone-reference
        mount: /tmp/clone-reference
        mode: ro

[fkorotkov]

I think in order to implement the self-cleanup mechanism we'll need to create a Kubernetes operator. We can call it expire. Such operator will delete objects from Kubernetes based on expire-timestamp. expire-timestamp will be updated in case of time based lifecycle.