Support CSI ephemeral volumes

[bentsherman]

Nextflow currently uses PersistentVolumeClaim to use any kind of persistent volume in a generic way, which is good.

However, there are also several types of ephemeral volumes:

• emptyDir
• configMap
• downwardAPI
• secret
• CSI ephemeral volume

Of these, Nextflow currently supports configMaps and secrets.

• emptyDir allows you to allocate local scratch storage in a k8s-friendly way, and there is an ancient PR (#1389) to support it in Nextflow
• downwardAPI allows you to pass node/pod information (e.g. hostname) to the pod via files, it is nice but can also be done via env / fieldPath which Nextflow supports
• CSI is basically a way to provision storage backed by third-party drivers. Some drivers can be persistent while others can be ephemeral. Secrets Store is a CSI ephemeral driver that allows you to inject secrets from something like Hashicorp Vault, Azure Keyvault, etc, which is more secure than using secrets.

These types of volumes don't fit in with the standard PVC model that Nextflow relies on, because you can't define them elsewhere and them reference them generically in the pod -- they're ephemeral! It only makes sense to define them in the pod itself. Therefore, Nextflow is already using the correct model with configMap and secret, and I think we should continue that pattern for the other ephemeral types. The most urgent one to add is csi, but I would support emptyDir as well. I will draft a PR for csi when I have some time.

[kanor1306]

Hi, just as a comment, there is another ephemeral volume type called "Generic Ephemeral Volumes" that essentially gives the capability of creating ephemeral volumes to any CSI driver capable of managing PVs and PVCs. This is documented in https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes (Beta since v1.19).

The main reason why this is very interesting is because it seems that the official aws-csi-ebs-driver will only support this approach for ephemeral volumes (or at least it is what the comments in that issue suggest)

Is there any chance that this will be supported in Nextflow even although it breaks a bit with the PVC approach (it requires to use the so called PVC templates)

[bentsherman]

@kanor1306 So you can provision an ephemeral CSI volume with this approach? If yes then maybe it would be better for Nextflow to use, otherwise we should consider it in a separate issue.

[kanor1306]

@bentsherman Well, it is an ephemeral volume, but not an ephemeral CSI volume.

My understanding is that the approach described in https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#csi-ephemeral-volumes that is also what is being implemented here is meant to be used with drivers that fully adhere to the CSI specification. Like this:

kind: Pod
...
spec:
  ...
  volumes:
    - name: my-csi-inline-vol
      csi:
        driver: inline.storage.kubernetes.io
        volumeAttributes:
          foo: bar

The problem (and I am assuming that this is something the Kubernetes developers realized at some point) is that out there there are many drivers that only adhere partially to the CSI specification, and some of them don't support the standard approach to ephemeral volumes, most notably the aws-ebs-csi-driver. In order to allow ephemeral in these drivers they came with the approach of making an inline declaration of an ephemeral volume claim described in https://kubernetes.io/docs/concepts/storage/ephemeral-volumes/#generic-ephemeral-volumes. Like this:

kind: Pod
...
spec:
  ...
  volumes:
    - name: scratch-volume
      ephemeral:
        volumeClaimTemplate:
          spec:
            accessModes: [ "ReadWriteOnce" ]
            storageClassName: "scratch-storage-class"
            resources:
              requests:
                storage: 1Gi

All this to say that you are right and this is probably for a different issue!