Support Ceph

[pothos]

Current situation The Ceph mount tool is not available and probably something else is missing (e.g., check kernel config).

Impact No Ceph for network storage, just NFS

Ideal future situation Ceph is supported

Implementation options FUSE vs kernel driver

Additional information

[mback2k]

I would prefer the kernel option since it seems like the required kernel modules are already in place. I guess just the mount helper tool mount.ceph is missing. I will try to compile it somewhere else and then test it on Flatcar manually.

[mback2k]

So it seems like Ceph itself is already supported by the kernel in Flatcar Linux. It is just the helper mount.ceph missing which is not a requirement for mounting according to https://docs.ceph.com/en/latest/cephfs/mount-using-kernel-driver/.

I was able to successfully mount CephFS via IP addresses without the helper. But taking the helper from Ubuntu's ceph-fs-common also worked.

[skydiablo]

@mback2k how do you do this? plain via CLI or a Ignition-YAML file? so i try to define a ignition file for such a mount, currently without success...

[ader1990]

I am also running Ceph using K8S (Rook Ceph) for the past two years already, working great so far. One needs to bump the ulimits, that is the only requirement to run Rook Ceph.

@skydiablo you can try it out using containers, by creating a container with adequate privilleges and the userspace libraries and workflows required.

[mback2k]

I was also running Ceph using Rook on Kubernetes. OS-level mounting was then done via systemd.mount files.

[skydiablo]

@mback2k may you can post your ".mount" file?

[skydiablo]

this is my current success:

storage:
  directories:
    - path: /mnt/cephfs_docker
      mode: 0755
systemd:
  units:
    - name: mnt-cephfs_docker.mount
      enabled: true
      contents: |
        [Unit]
        Description=Mount CephFS for Docker
        After=network-online.target
        Wants=network-online.target
        Before=docker.service
        Requires=network-online.target

        [Mount]
        What=IP_NODE1:6789,IP_NODE2:6789,IP_NODE3:6789:/OPTIONAL_FOLDER_ON_CPEH_FS/
        Where=/mnt/cephfs_docker
        Type=ceph
        Options=name=CEPH_USERNAME,secret=KEYRING,_netdev,noatime,mds_namespace=POOL_NAME,fsid=CEPH_CLUSTER_FSID,acl

        [Install]
        WantedBy=multi-user.target

i have also tried to use to define a "secretfile", but the kernel-client doesnt know this option. maybe someone will help this.

to create a new user:

ceph auth get-or-create client.CEPH_USERNAME mon 'allow r' osd 'allow rwx pool=POOL_NAME' mds "allow *"

to get the secret

ceph auth get client.CEPH_USERNAME

to create your cephfs, plz use some other research ;)