Longhorn directory not persistent

[SanekP]

What steps did you take and what happened: Installed Elemental extension. helm upgrade --create-namespace -n cattle-elemental-system --install elemental-operator-crds oci://registry.suse.com/rancher/elemental-operator-crds-chart helm upgrade --create-namespace -n cattle-elemental-system --install elemental-operator oci://registry.suse.com/rancher/elemental-operator-chart OS Version Channel elemental-channel which is registry.suse.com/rancher/elemental-channel:1.4.2 contains ManagedOSVersion v2.0.2. Install this version from fresh Seed Image or upgrade to it through Update Groups. [A clear and concise description of what the bug is.] Installation does not contain persistent path /var/lib/longhorn.

What did you expect to happen: As it stated in documentation https://elemental.docs.rancher.com/installation path /var/lib/longhorn must be persistent. So I expected that it has mount to persistent partition as it was in previous versions e.g. v1.2.3. m# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0 1.5G 1 loop / sda 8:0 0 465.8G 0 disk ├─sda1 8:1 0 1M 0 part ├─sda2 8:2 0 64M 0 part /oem ├─sda3 8:3 0 4G 0 part ├─sda4 8:4 0 8G 0 part /run/initramfs/cos-state └─sda5 8:5 0 453.7G 0 part /var/lib/calico /var/lib/cni /var/lib/NetworkManager /var/lib/kubelet /var/lib/rancher /var/lib/elemental /var/log /usr/libexec /root /opt /home /etc/cni /etc/iscsi /etc/ssh /etc/rancher /etc/systemd /usr/local sdb 8:16 1 0B 0 disk sr0 11:0 1 1024M 0 rom

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Elemental release version (use cat /etc/os-release): m# cat /etc/os-release NAME="SLE Micro" VERSION="5.5" VERSION_ID="5.5" PRETTY_NAME="SUSE Linux Enterprise Micro for Rancher 5.5" ID="sle-micro-rancher" ID_LIKE="suse" ANSI_COLOR="0;32" CPE_NAME="cpe:/o:suse:sle-micro-rancher:5.5" IMAGE_TAG="2.0.2-4.2.16" IMAGE="registry.suse.com/suse/sle-micro/base-5.5:2.0.2-4.2.16" TIMESTAMP=20240131165042 GRUB_ENTRY_NAME="SLE Micro" IMAGE_REPO="registry.suse.com/suse/sle-micro/5.5" IMAGE_TAG="2.0.2-4.2.20" IMAGE="registry.suse.com/suse/sle-micro/5.5:2.0.2-4.2.20" TIMESTAMP=20240131165621 GRUB_ENTRY_NAME="SLE Micro"
• Rancher version: v2.8.1
• Kubernetes version (use kubectl version): m# kubectl version WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short. Use --output=yaml|json to get the full version. Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.8+k3s2", GitCommit:"02fcbd1f57f0bc0ca1dc68f98cfa0e7d3b008225", GitTreeState:"clean", BuildDate:"2023-12-07T02:48:20Z", GoVersion:"go1.20.11", Compiler:"gc", Platform:"linux/amd64"} Kustomize Version: v5.0.1 Server Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.8+k3s2", GitCommit:"02fcbd1f57f0bc0ca1dc68f98cfa0e7d3b008225", GitTreeState:"clean", BuildDate:"2023-12-07T02:48:20Z", GoVersion:"go1.20.11", Compiler:"gc", Platform:"linux/amd64"}
• Cloud provider or hardware configuration: test

[frelon]

Hi @SanekP!

Looks like that documentation is outdated, /var/lib/longhorn was removed as a default persistent path since there was problems trying to use longhorn with LVM when that path was already a persistent mount.

I would recommend following the https://elemental.docs.rancher.com/customizing/#adding-extra-lvm-volume-group-disks-during-the-installation guide to add a custom rootfs mount layout like the following:

name: "Custom Rootfs Layout Settings"
stages:
  rootfs.after:
    - if: '[ ! -f "/run/elemental/recovery_mode" ]'
      name: "Layout configuration"
      environment_file: /run/elemental/mount-layout.env
      environment:
        OVERLAY: "tmpfs:25%"
        RW_PATHS: "/var /etc /srv"
        PERSISTENT_STATE_BIND: "true"
        PERSISTENT_STATE_PATHS: >-
          /etc/systemd
          /etc/rancher
          /etc/ssh
          /etc/iscsi
          /etc/cni
          /home
          /opt
          /root
          /usr/libexec
          /usr/local
          /var/log
          /var/lib/elemental
          /var/lib/longhorn
          /var/lib/rancher
          /var/lib/kubelet
          /var/lib/NetworkManager
          /var/lib/cni
          /var/lib/calico

[SanekP]

This all looks so overcomplicated! I don't want to make deep customization, I just want to make /var/lib/longhorn path persistent (using existing COS_PERSISTENT partition). Is there somewhere an original of this rootfs mount layout? I mean something in RW_PATHS could change or I can miss some of PERSISTENT_STATE_PATHS.

[kkaempf]

Agreed @SanekP. I've created #1239 to address your concern.

[SanekP]

Also it is not expected after upgrading from previous OS version. Maybe it worth to create a Troubleshooting upgrade article how to mitigate this issue.

As I understand for the new installation I need somehow to put content from frelon to the /iso-config folder of ISO file. Is it possible to do same from MachineRegistration resource spec? It would be nice to download already ready ISO file.