Filesystem doesn't resize on RPI4 and alpine flavor

[Ludea]

Kairos version:

tested version : 3.1.2 and 3.1.3 flavor: alpine

CPU architecture, OS, and Version:

Rp4 aarch64

Describe the bug

Filesystem is not resizing Partition resizing works as expected

To Reproduce

• Do a fresh install install of Kairos on sd card
• Boot Rpi
• run df -h to check the COS_PERSISTENT disk

Expected behavior

Filesystem expand all available free space

Logs

df -h:

Filesystem                         Size  Used Avail Use% Mounted on
 /dev/disk/by-label/COS_ACTIVE      2.0G  697M  1.2G  38% /
 /dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
 /dev/disk/by-label/COS_PERSISTENT   55M  1.9M   50M   4% /usr/local
 overlay                            2.0G  180K  2.0G   1% /var
 overlay                            2.0G  180K  2.0G   1% /etc
 overlay                            2.0G  180K  2.0G   1% /srv
 devtmpfs                            10M     0   10M   0% /dev
 tmpfs                              3.9G  544K  3.9G   1% /run
 shm                                3.9G     0  3.9G   0% /dev/shm
 tmpfs                              3.9G     0  3.9G   0% /tmp
 /dev/mmcblk0p2                     5.9G  1.4G  4.3G  25% /run/initramfs/cos-state
 tmpfs                              2.0G     0  2.0G   0% /run/overlay

dmesg:

[    4.651890] udevd[995]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:1': No such file or directory
 [    4.652275] udevd[997]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 179:0': No such file or directory
 [    4.652795] udevd[996]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:10': No such file or directory
 [    4.653498] udevd[998]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:0': No such file or directory
 [    4.653902] udevd[999]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:11': No such file or directory
 [    4.654311] udevd[1000]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:12': No such file or directory
 [    4.654535] udevd[1001]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:13': No such file or directory
 [    4.677703] NET: Registered PF_PACKET protocol family
 [    4.683017] udevd[1007]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 1:7': No such file or directory
 [    4.686944] udevd[1009]: failed to execute '/usr/libexec/nfsrahead' '/usr/libexec/nfsrahead 7:0': No such file or directory 

immucore.log:

1970-01-01T00:00:11Z INF Immucore commit=none compiled with=go1.22.5 version=v0.4.2
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF Booting on active/passive/recovery.
 1970-01-01T00:00:11Z INF 1.
  <init> (background: false) (weak: false) (run: false)
 2.
  <lvm-activation> (background: false) (weak: false) (run: false)
  <mount-state> (background: false) (weak: false) (run: false)
  <mount-tmpfs> (background: false) (weak: false) (run: false)
  <create-sentinel> (background: false) (weak: false) (run: false)
 3.
  <discover-state> (background: false) (weak: false) (run: false)
  <upgrade-kcrypt> (background: false) (weak: false) (run: false)
 4.
  <mount-root> (background: false) (weak: false) (run: false)
 5.
  <mount-oem> (background: false) (weak: false) (run: false)
 6.
  <rootfs-hook> (background: false) (weak: false) (run: false)
  <unlock-all> (background: false) (weak: false) (run: false)
 7.
  <load-config> (background: false) (weak: false) (run: false)
 8.
  <mount-base-overlay> (background: false) (weak: false) (run: false)
  <custom-mount> (background: false) (weak: false) (run: false)
 9.
  <overlay-mount> (background: false) (weak: false) (run: false)
 10.
  <mount-bind> (background: false) (weak: false) (run: false)
 11.
  <write-fstab> (background: false) (weak: false) (run: false)
 12.
  <initramfs-hook> (background: false) (weak: false) (run: false)
 
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:11Z INF creating a runtime
 1970-01-01T00:00:11Z INF Setting sentinel file to=active_mode
 1970-01-01T00:00:11Z INF detecting boot state
 1970-01-01T00:00:12Z INF mount done options=["rw"] type=tmpfs what=tmpfs where=/tmp
 1970-01-01T00:00:12Z INF mount done options=["ro"] type=ext3 what=/dev/disk/by-label/COS_STATE where=/sysroot/run/initramfs/cos-state
 1970-01-01T00:00:13Z INF mount done options=["ro","suid","dev","exec","async"] type=ext4 what=/dev/disk/by-label/COS_ACTIVE where=/sysroot
 1970-01-01T00:00:13Z INF creating a runtime
 1970-01-01T00:00:13Z INF detecting boot state
 1970-01-01T00:00:14Z INF mount done options=["rw","suid","dev","exec","async"] type=ext3 what=/dev/disk/by-label/COS_OEM where=/sysroot/oem
 1970-01-01T00:00:14Z INF Running rootfs stage
 1970-01-01T00:00:14Z INF triggering udev to populate disk info
 1970-01-01T00:00:21Z INF mount done options=["rw"] type=ext4 what=/dev/disk/by-label/COS_PERSISTENT where=/sysroot/usr/local
 1970-01-01T00:00:21Z WRN
 1970-01-01T00:00:21Z WRN executing mount callback error="mkdir /sysroot/snap: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/snap.bind where=/sysroot/snap
 1970-01-01T00:00:21Z ERR error="mkdir /sysroot/snap: read-only file system"
 1970-01-01T00:00:24Z WRN executing mount callback error="mkdir /sysroot/usr/share/pki: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/usr-share-pki-trust.bind where=/sysroot/usr/share/pki/trust
 1970-01-01T00:00:24Z ERR error="mkdir /sysroot/usr/share/pki: read-only file system"
 1970-01-01T00:00:24Z WRN executing mount callback error="mkdir /sysroot/usr/share/pki: read-only file system" options=["bind"] type=overlay what=/sysroot/usr/local/.state/usr-share-pki-trust-anchors.bind where=/sysroot/usr/share/pki/trust/anchors
 1970-01-01T00:00:24Z ERR error="mkdir /sysroot/usr/share/pki: read-only file system"
 1970-01-01T00:00:24Z WRN error="3 errors occurred:\n\t* mkdir /sysroot/snap: read-only file system\n\t* mkdir /sysroot/usr/share/pki: read-only file system\n\t* mkdir /sysroot/usr/share/pki: read-only file system\n\n"
 1970-01-01T00:00:24Z INF Running initramfs stage
 2006-08-14T08:35:02Z INF 1.
  <init> (background: false) (weak: false) (run: false)
 2.
  <create-sentinel> (background: false) (weak: false) (run: true)
  <mount-tmpfs> (background: false) (weak: false) (run: true)
  <lvm-activation> (background: false) (weak: false) (run: true)
  <mount-state> (background: false) (weak: false) (run: true)
 3.
  <upgrade-kcrypt> (background: false) (weak: false) (run: true)
  <discover-state> (background: false) (weak: false) (run: true)
 4.
  <mount-root> (background: false) (weak: false) (run: true)
 5.
  <mount-oem> (background: false) (weak: false) (run: true)
 6.
  <rootfs-hook> (background: false) (weak: false) (run: true)
  <unlock-all> (background: false) (weak: false) (run: true)
 7.
  <load-config> (background: false) (weak: false) (run: true)
 8.
  <custom-mount> (background: false) (weak: false) (run: true)
  <mount-base-overlay> (background: false) (weak: false) (run: true)
 9.
  <overlay-mount> (background: false) (weak: false) (run: true)
 10.
  <mount-bind> (error: 3 errors occurred:
         * mkdir /sysroot/snap: read-only file system
         * mkdir /sysroot/usr/share/pki: read-only file system
         * mkdir /sysroot/usr/share/pki: read-only file system
 
 ) (background: false) (weak: false) (run: true)
 11.
  <write-fstab> (background: false) (weak: false) (run: true)
 12.
  <initramfs-hook> (background: false) (weak: false) (run: true)

Additional context

Itxaka try to reproduce issue on his Rpi, but FS resize as expected. With Tumbleweed flavor, no issue, partition and fs resizing work as expected

[Itxaka]

going to retry and put the logs here to see.

[Itxaka]

just intalled from scratch and cant reproduce:

Disk /dev/sda: 115.1 GiB, 123589361664 bytes, 241385472 sectors
Disk model: 3.2 Gen 1       
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 33553920 bytes
Disklabel type: gpt
Disk identifier: 85800D3E-DE8D-4BE1-8D6A-FDE3987F8CD8

Device        Start       End   Sectors   Size Type
/dev/sda1      2048    198655    196608    96M EFI System
/dev/sda2    198656  12896255  12697600   6.1G Linux filesystem
/dev/sda3  12896256  21497855   8601600   4.1G Linux filesystem
/dev/sda4  21497856  21628927    131072    64M Linux filesystem
/dev/sda5  21628928 241385436 219756509 104.8G Linux filesystem
localhost:~$ df -h
Filesystem                         Size  Used Avail Use% Mounted on
/dev/disk/by-label/COS_ACTIVE      2.0G  697M  1.2G  38% /
/dev/disk/by-label/COS_OEM          55M   32K   52M   1% /oem
/dev/disk/by-label/COS_PERSISTENT   99G  1.8M   93G   1% /usr/local
overlay                            2.0G  176K  2.0G   1% /var
overlay                            2.0G  176K  2.0G   1% /etc
overlay                            2.0G  176K  2.0G   1% /srv
devtmpfs                            10M     0   10M   0% /dev
tmpfs                              3.9G  608K  3.9G   1% /run
efivarfs                           128K   920  128K   1% /sys/firmware/efi/efivars
shm                                3.9G     0  3.9G   0% /dev/shm
tmpfs                              3.9G     0  3.9G   0% /tmp
/dev/sda2                          5.9G  1.4G  4.3G  25% /run/initramfs/cos-state
tmpfs                              2.0G     0  2.0G   0% /run/overlay

localhost:~$ cat /etc/kairos-release 
KAIROS_PRETTY_NAME="kairos-standard-alpine-3.19 v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_IMAGE_REPO="quay.io/kairos/alpine:3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1-k3s1"
KAIROS_IMAGE_LABEL="3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1-k3s1"
KAIROS_REGISTRY_AND_ORG="quay.io/kairos"
KAIROS_BUG_REPORT_URL="https://github.com/kairos-io/kairos/issues"
KAIROS_SOFTWARE_VERSION="v1.31.1+k3s1"
KAIROS_ID="kairos"
KAIROS_VERSION="v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_FLAVOR="alpine"
KAIROS_FAMILY="alpine"
KAIROS_SOFTWARE_VERSION_PREFIX="k3s"
KAIROS_NAME="kairos-standard-alpine-3.19"
KAIROS_VERSION_ID="v3.2.1-7-ga068314-v1.31.1-k3s1"
KAIROS_MODEL="rpi4"
KAIROS_HOME_URL="https://github.com/kairos-io/kairos"
KAIROS_ID_LIKE="kairos-standard-alpine-3.19"
KAIROS_FLAVOR_RELEASE="3.19"
KAIROS_VARIANT="standard"
KAIROS_TARGETARCH="arm64"
KAIROS_RELEASE="v3.2.1-7-ga068314"
KAIROS_GITHUB_REPO="kairos-io/kairos"
KAIROS_ARTIFACT="kairos-alpine-3.19-standard-arm64-rpi4-v3.2.1-7-ga068314-k3sv1.31.1+k3s1"

Stage is correctly executed

1970-01-01T00:00:20Z INF Processing stage step 'RPI configs.Grow persistent'. ( commands: 0, files: 0, ... )
1970-01-01T00:00:20Z DBG Stage: schema.Stage{
  If: "[ ! -f /run/cos/recovery_mode ] && [ ! -f /run/cos/live_mode ] && [ -f \"/sys/firmware/devicetree/base/model\" ] && grep -i \"Raspberry Pi 4\" \"/sys/firmware/devicetree/base/model\"",
  Name: "Grow persistent",
  Layout: schema.Layout{
    Device: &schema.Device{
      Label: "COS_PERSISTENT",
    },
    Expand: &schema.Expand{},
  },
}
1970-01-01T00:00:20Z DBG Using label COS_PERSISTENT for layout expansion
1970-01-01T00:00:20Z DBG Output of udevadm settle: 
1970-01-01T00:00:20Z DBG Got device /dev/sda for label COS_PERSISTENT
1970-01-01T00:00:21Z INF Extending last partition to max space
1970-01-01T00:00:21Z DBG Expanding partition 5 up to 0 sectors
1970-01-01T00:00:22Z DBG Output of growpart: CHANGED: partition=5 start=21628928 old: size=131072 end=21759999 new: size=219756510 end=241385437

1970-01-01T00:00:22Z DBG Trying to reread the partition table of /dev/sda (try number 1)
1970-01-01T00:00:22Z DBG output of partprobe: 
1970-01-01T00:00:22Z DBG Output of sync: 
1970-01-01T00:00:22Z DBG Trying to resize filesystem for device /dev/sda5
1970-01-01T00:00:22Z DBG Found filesystem ext3
 for device /dev/sda5
1970-01-01T00:00:22Z DBG Output from running e2fsck e2fsck 1.47.0 (5-Feb-2023)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
COS_PERSISTENT: 11/16384 files (0.0% non-contiguous), 9525/65536 blocks

1970-01-01T00:01:57Z DBG Output from running resize2fs resize2fs 1.47.0 (5-Feb-2023)
Resizing the filesystem on /dev/sda5 to 109878252 (1k) blocks.
The filesystem on /dev/sda5 is now 109878252 (1k) blocks long.


1970-01-01T00:01:57Z DBG Trying to reread the partition table of /dev/sda (try number 1)
1970-01-01T00:01:57Z DBG output of partprobe: 
1970-01-01T00:01:57Z DBG Output of sync: 

I have no idea whats going on @Ludea Please retry from scratch and run the Kairos entry adding rd.immucore.debug to the cmdline and then paste the /run/immucore/immucore.log and /run/immucore/rootfs_stage.log full logs to see whats going on.

[Ludea]

I attach some logs from immucore.log and rootfs_stage.log

[Ludea]

Partition is resizing (check with lsblk) Fs is not resizing (check with df - h) I see https://github.com/kairos-io/kairos/issues/2128 May I wait this release?

[jimmykarily]

@Ludea 's output:

Got device /dev/mmcblk0 for label COS_PERSISTENT

@Itxaka 's :

Got device /dev/sda for label COS_PERSISTENT

Maybe relevant to this? https://github.com/kairos-io/kairos-agent/commit/84d87b3a5dc18b835a4098b6acef16b62e573e32

[Itxaka]

Difference between SD and USB boot device I guess. I also tried with SD card

[jimmykarily]

Let's wait for the upcoming release 3.2.2 (which includes the linked fix) so @Ludea can give it another try.

[Ludea]

I cannot test the fix :s I resize myself the fs and deploy all my infra. But thanks for fixing it ! In case I re flash kairos, will happy to try.

[jimmykarily]

All good then. I'll close this but let us know if you face any issues in the future.