volsync icon indicating copy to clipboard operation
volsync copied to clipboard

Published 20 hours ago • verified publisher icon backube

Allow volumes and volume mounts to be specified

Open mimiteto opened this issue 1 year ago • 2 comments

Describe the feature you'd like to have. It would be nice to have the option to mount user specified secrets, configmaps (and others) on the replication pods.

What is the value to the end user? (why is it a priority?) As an end user I would be able to use ssh certificates with the rclone mover (when using rclone with sftp). Currently certificates in rclone can be provided only as file paths in their config. There are possibly other cases where the ability to provide similar configurations is beneficial.

How will we know we have a good solution? (acceptance criteria)

Additional context

mimiteto avatar Oct 16 '24 13:10 mimiteto

We could possibly do something similar to what we did with restic with customCA (that can be specified in a configmap or secret).

Is there a common way of passing this information to rclone, and would this give you what you need? A quick look suggests rclone has a --ca-cert option, but not sure if it works for non HTTPS backends).

The issue with simply mounting things to mover pods is we need to also specify paths so that you can reference them, and also potentially tell rclone where to look for them. I'd prefer to have defined parameters we pass to rclone if possible.

tesshuflower avatar Oct 18 '24 14:10 tesshuflower

On the topic of rclone - the flag I need is --sftp-pubkey-file. I proposed a code change so I don't need to mount anything beyond what is currently available in volsync.

On the topic of mounting user specified volumes - I believe this will make volsync more flexible. Currently you are handling each case separately which needs issues like the current one. Allowing user specified volumes to be mounted will remove the need to open new issues. I believe you can just add user specified volumes to a common dir (e.g. /mnt or /mount). Let's look at the current replication source

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: user-sync-data-0
  namespace: sync-data
spec:
  rclone:
    copyMethod: Clone
    moverSecurityContext:
      fsGroup: 1000
      runAsGroup: 1000
      runAsUser: 1000
    rcloneConfig: volsync-user-sync-local
    rcloneConfigSection: ssh
    rcloneDestPath: /backups/p/sync-data/user-sync-data-user-sync-0
  sourcePVC: user-sync-data-user-sync-0
  trigger:
    schedule: 0 6 * * *

Which leads to the following job

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: "2024-10-18T15:24:53Z"
  generation: 1
  labels:
    app.kubernetes.io/created-by: volsync
    volsync.backube/cleanup: d56bcd01-3649-4629-8a1d-692345b5be2e
  name: volsync-rclone-src-user-sync-data-0
  namespace: sync-data
  ownerReferences:
  - apiVersion: volsync.backube/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicationSource
    name: user-sync-data-0
    uid: d56bcd01-3649-4629-8a1d-692345b5be2e
  resourceVersion: "11063059"
  uid: bad912e6-0b04-477b-9652-957faa264ee6
spec:
  backoffLimit: 2
  completionMode: NonIndexed
  manualSelector: false
  parallelism: 1
  podReplacementPolicy: TerminatingOrFailed
  selector:
    matchLabels:
      batch.kubernetes.io/controller-uid: bad912e6-0b04-477b-9652-957faa264ee6
  suspend: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/created-by: volsync
        batch.kubernetes.io/controller-uid: bad912e6-0b04-477b-9652-957faa264ee6
        batch.kubernetes.io/job-name: volsync-rclone-src-user-sync-data-0
        controller-uid: bad912e6-0b04-477b-9652-957faa264ee6
        job-name: volsync-rclone-src-user-sync-data-0
      name: volsync-rclone-src-user-sync-data-0
    spec:
      containers:
      - command:
        - /bin/bash
        - -c
        - /mover-rclone/active.sh
        env:
        - name: RCLONE_CONFIG
          value: /rclone-config/rclone.conf
        - name: RCLONE_DEST_PATH
          value: /backups/p/sync-data/user-sync-data-user-sync-0
        - name: DIRECTION
          value: source
        - name: MOUNT_PATH
          value: /data
        - name: RCLONE_CONFIG_SECTION
          value: ssh
        - name: PRIVILEGED_MOVER
          value: "0"
        image: quay.io/backube/volsync:0.10.0
        imagePullPolicy: IfNotPresent
        name: rclone
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: data
        - mountPath: /rclone-config/
          name: rclone-secret
        - mountPath: /tmp
          name: tempdir
      dnsPolicy: ClusterFirst
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsUser: 1000
      serviceAccount: volsync-src-user-sync-data-0
      serviceAccountName: volsync-src-user-sync-data-0
      terminationGracePeriodSeconds: 30
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: volsync-user-sync-data-0-src
      - name: rclone-secret
        secret:
          defaultMode: 384
          secretName: volsync-user-sync-local
      - emptyDir:
          medium: Memory
        name: tempdir

Allowing users to specify volumes like:

apiVersion: volsync.backube/v1alpha1
kind: ReplicationSource
metadata:
  name: user-sync-data-0
  namespace: sync-data
spec:
  rclone:
    copyMethod: Clone
    moverSecurityContext:
      fsGroup: 1000
      runAsGroup: 1000
      runAsUser: 1000
    rcloneConfig: volsync-user-sync-local
    rcloneConfigSection: ssh
    rcloneDestPath: /backups/p/sync-data/user-sync-data-user-sync-0
  sourcePVC: user-sync-data-user-sync-0
  trigger:
    schedule: 0 6 * * *
  moverVolumes:
    - name: my-mount
      persistentVolumeClaim:
        claimName: my-pvc
    - name: my-credentials
      secret:
        secretName: secret-credentials

and then producing job like

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: "2024-10-18T15:24:53Z"
  generation: 1
  labels:
    app.kubernetes.io/created-by: volsync
    volsync.backube/cleanup: d56bcd01-3649-4629-8a1d-692345b5be2e
  name: volsync-rclone-src-user-sync-data-0
  namespace: sync-data
  ownerReferences:
  - apiVersion: volsync.backube/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicationSource
    name: user-sync-data-0
    uid: d56bcd01-3649-4629-8a1d-692345b5be2e
  resourceVersion: "11063059"
  uid: bad912e6-0b04-477b-9652-957faa264ee6
spec:
  backoffLimit: 2
  completionMode: NonIndexed
  manualSelector: false
  parallelism: 1
  podReplacementPolicy: TerminatingOrFailed
  selector:
    matchLabels:
      batch.kubernetes.io/controller-uid: bad912e6-0b04-477b-9652-957faa264ee6
  suspend: false
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/created-by: volsync
        batch.kubernetes.io/controller-uid: bad912e6-0b04-477b-9652-957faa264ee6
        batch.kubernetes.io/job-name: volsync-rclone-src-user-sync-data-0
        controller-uid: bad912e6-0b04-477b-9652-957faa264ee6
        job-name: volsync-rclone-src-user-sync-data-0
      name: volsync-rclone-src-user-sync-data-0
    spec:
      containers:
      - command:
        - /bin/bash
        - -c
        - /mover-rclone/active.sh
        env:
        - name: RCLONE_CONFIG
          value: /rclone-config/rclone.conf
        - name: RCLONE_DEST_PATH
          value: /backups/p/sync-data/user-sync-data-user-sync-0
        - name: DIRECTION
          value: source
        - name: MOUNT_PATH
          value: /data
        - name: RCLONE_CONFIG_SECTION
          value: ssh
        - name: PRIVILEGED_MOVER
          value: "0"
        image: quay.io/backube/volsync:0.10.0
        imagePullPolicy: IfNotPresent
        name: rclone
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /data
          name: data
        - mountPath: /rclone-config/
          name: rclone-secret
        - mountPath: /tmp
          name: tempdir
        - mountPath: /mnt/my-mount
          name: u-my-mount
        - mountPath: /mnt/my-credentials
          name: u-my-credentials
      dnsPolicy: ClusterFirst
      restartPolicy: Never
      schedulerName: default-scheduler
      securityContext:
        fsGroup: 1000
        runAsGroup: 1000
        runAsUser: 1000
      serviceAccount: volsync-src-user-sync-data-0
      serviceAccountName: volsync-src-user-sync-data-0
      terminationGracePeriodSeconds: 30
      volumes:
      - name: data
        persistentVolumeClaim:
          claimName: volsync-user-sync-data-0-src
      - name: rclone-secret
        secret:
          defaultMode: 384
          secretName: volsync-user-sync-local
      - emptyDir:
          medium: Memory
        name: tempdir
      - name: u-my-mount
        persistentVolumeClaim:
          claimName: my-pvc
      - name: u-my-credentials
        secret:
          secretName: secret-credentials

with added prefix (u- in my example) to avoid collision the the current existing volume names.

Users then can reliably guess the path they need for their configs.

mimiteto avatar Oct 18 '24 15:10 mimiteto

@tesshuflower I think @mimiteto suggestion here would be a very good solution to this problem and the past discussions and PRs on this issue. Do you agree?

onedr0p avatar Sep 24 '25 14:09 onedr0p

I may try giving it a shot over the weekend if this solution seems good to you, guys!

mimiteto avatar Sep 24 '25 14:09 mimiteto

We're having an internal meeting next week, and I'll bring this up there.

This sounds like a good idea - however we do have some considerations when it comes to adding these flexible do-anything options:

  • More complicated spec and instructions for users
  • I'm unsure how this benefits other movers at this time? Rclone has the most configuration options via the RCLONE_ env vars so I do see how it can help there more easily.
  • The biggest concern for me is really that any user who can create volsync CRs in a namespace can now mount any secret in the namespace to their mover, so I'd like to discuss further with others.

tesshuflower avatar Sep 25 '25 13:09 tesshuflower

My use-case for this feature circle backs to my previous issues and PRs, being able to mount in a NFS volume for restic to use.

...
  moverVolumes:
    - name: repository
      nfs:
        path: /tank/restic
        server: nas.internal
...

When deploying on-prem having the ability to reduce external dependencies (e.g. minio/whatever) for s3 is always a nice thing, especially now that minio decided to completely screw people over who use it for free.

onedr0p avatar Sep 25 '25 21:09 onedr0p

We've made the decision to go ahead with the implementation as suggested above, we will do this for PVCs and Secrets as a start.

tesshuflower avatar Oct 01 '25 14:10 tesshuflower

/close

Implemented in #1778

tesshuflower avatar Nov 05 '25 15:11 tesshuflower

@tesshuflower: Closing this issue.

In response to this:

/close

Implemented in #1778

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

openshift-ci[bot] avatar Nov 05 '25 15:11 openshift-ci[bot]