9p cache seems broken on alpine build

[casatir]

First of all, thanks a lot for this great project!

I have built the alpine 9p filesystem and the vm state using build.sh then build-state.js in tools/docker/alpine/. After loading them in the browser, each 9p file access generates a network (http) request so the cache=loose kernel parameter seems to be ignored.

Moreover, I don't know if it's related but the mount output contains the line

host9p on / type 9p (rw,relatime,cache=f,access=client,trans=virtio)

should it not be cache=loose (or loose alone, I don't know) ?

Just to mention that the 9p cache is working great when using https://copy.sh/v86/?profile=archlinux and loose appear in the 9p mount options.

Thanks for your help.

[copy]

The code for this is here: https://github.com/alpinelinux/mkinitfs/blob/54bae0769080710e0769370a7bf3e0e158eec152/initramfs-init.in#L740

The flags looks fine to me, maybe it's something weird with busybox's mount?

[SuperMaxusa]

The flags looks fine to me, maybe it's something weird with busybox's mount?

I think so, it doesn't work on "virt" LiveCD either: mount -t 9p -o cache=loose host9p /mnt.

UPD: This is not a busybox's bug, mount flags are set correctly for mount() syscall:

strace log

execve("/bin/mount", ["mount", "-t", "9p", "-o", "trans=virtio,cache=loose", "host9p", "/mnt"], 0xbffb9f70 /* 12 vars */) = 0
set_thread_area({entry_number=-1, base_addr=0xb7f7cc64, limit=0x0fffff, seg_32bit=1, contents=0, read_exec_only=0, limit_in_pages=1, seg_not_present=0, useable=1}) = 0 (entry_number=6)
set_tid_address(0xb7f7ccfc)             = 1869
brk(NULL)                               = 0x1efe000
brk(0x1f00000)                          = 0x1f00000
mmap2(0x1efe000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x1efe000
mprotect(0xb7f7a000, 4096, PROT_READ)   = 0
mprotect(0x4b9000, 4096, PROT_READ)     = 0
execve("/bin/busybox", ["/bin/busybox", "mount", "-t", "9p", "-o", "trans=virtio,cache=loose", "host9p", "/mnt"], 0xbfe3cfb4 /* 12 vars */) = 0
set_thread_area({entry_number=-1, base_addr=0xb7f39c64, limit=0x0fffff, seg_32bit=1, contents=0, read_exec_only=0, limit_in_pages=1, seg_not_present=0, useable=1}) = 0 (entry_number=6)
set_tid_address(0xb7f39cfc)             = 1869
brk(NULL)                               = 0x23df000
brk(0x23e1000)                          = 0x23e1000
mmap2(0x23df000, 4096, PROT_NONE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x23df000
mprotect(0xb7f37000, 4096, PROT_READ)   = 0
mprotect(0x521000, 8192, PROT_READ)     = 0
getuid32()                              = 0
getuid32()                              = 0
geteuid32()                             = 0
statx(AT_FDCWD, "host9p", AT_STATX_SYNC_AS_STAT|AT_NO_AUTOMOUNT, STATX_BASIC_STATS, 0xbfabc5c0) = -1 ENOENT (No such file or directory)
mount("host9p", "/mnt", "9p", MS_SILENT, "trans=virtio,cache=loose") = 0
exit_group(0)                           = ?

Just to mention that the 9p cache is working great when using https://copy.sh/v86/?profile=archlinux and loose appear in the 9p mount options.

The kernel version in the archlinux profile is a older than in Alpine (5.19.7 vs 6.12.28), so I would recommend testing a current Debian or Void version to see if it's a kernel bug or not.

[casatir]

Thanks a lot for this answers. Booting the Void 32 bits live iso from https://voidlinux.org/download/#i686 in https://copy.sh/v86/?profile=custom&m=512&vram=32 shows

Looks like a kernel bug?

[casatir]

Just to mention that all cache options give other strange values. cache=mmap gives cache=5, and cache=fscache gives cache=8f.

[SuperMaxusa]

Tested 6 versions of Alpine Linux (v86 version: https://github.com/copy/v86/commit/b2a2b405e274bbfff9372f0d150b4eb1e7d47c0b):

Version	Kernel version	cache=loose works?
3.18.3 (iso)	6.1.43	Yes
3.18.5 (iso)	6.1.64	Yes
3.19.1 (iso)	6.6.14	No
3.19.5 (iso)	6.6.69	No
3.20.6 (iso)	6.6.76	No
3.21.3 (iso)	6.12.13	No

It seems that kernel versions 6.6+ have some breaking changes on 9pfs. We had a bug with that branch earlier: https://github.com/copy/v86/issues/996.

I haven't learn (virtio-)9p in detail yet, is the cache mode set only for v86 or Linux?

[casatir]

Thanks for your investigation. Installing 3.18.5 as a temporary workaround is fine for me but I get the following error

[   15.083049] Mounting root...
 * Mounting root: mount: mounting host9p on /sysroot failed: No such device
[   27.795942] Mounting root: failed. 
failed. 
initramfs emergency recovery shell launched. Type 'exit' to continue boot

even following the advice here (installing mkinitfs from edge).

I haven't learn (virtio-)9p in detail yet, is the cache mode set only for v86 or Linux?

To me, the cache mode prevent Linux from asking v86 to ask already fetched files to the server. Don't know if this answers your question.

[SuperMaxusa]

Installing 3.18.5 as a temporary workaround is fine for me but I get the following error <...> even following the advice here (installing mkinitfs from edge).

I tried replacing edge to v3.19 and moving this step before apk add <...> linux-$KERNEL and it works for me:

Full Dockerfile:

FROM docker.io/i386/alpine:3.18.5

ENV KERNEL=virt
ENV ADDPKGS=

# Installing newer mkinitfs
RUN apk add mkinitfs --no-cache --allow-untrusted --repository https://dl-cdn.alpinelinux.org/alpine/v3.19/main

RUN apk add openrc alpine-base agetty alpine-conf linux-$KERNEL linux-firmware-none $ADDPKGS

RUN sed -i 's/getty 38400 tty1/agetty --autologin root tty1 linux/' /etc/inittab
RUN echo 'ttyS0::respawn:/sbin/agetty --autologin root -s ttyS0 115200 vt100' >> /etc/inittab
RUN echo "root:" | chpasswd

RUN setup-hostname localhost

# Adding networking.sh script
RUN echo -e "rmmod ne2k-pci && modprobe ne2k-pci\nrmmod virtio-net && modprobe virtio-net\nhwclock -s\nsetup-interfaces -a -r" > /root/networking.sh && chmod +x /root/networking.sh

# https://wiki.alpinelinux.org/wiki/Alpine_Linux_in_a_chroot#Preparing_init_services
RUN for i in devfs dmesg mdev hwdrivers; do rc-update add $i sysinit; done
RUN for i in hwclock modules sysctl hostname syslog bootmisc; do rc-update add $i boot; done
RUN rc-update add killprocs shutdown

# Generate initramfs with 9p modules
RUN mkinitfs -F "base virtio 9p" $(cat /usr/share/kernel/$KERNEL/kernel.release)

[casatir]

Great, it's working with 3.18.5! And also 3.18.6 by the way. Good workaround, thanks!

[casatir]

I was reading https://docs.kernel.org/filesystems/9p.html and found this:

Converting masks to hexadecimal gives exactly the values we observed:

shortcut	mask (bits)	mask (hex)
mmap	00000101	5
loose	00001111	f
fscache	10001111	8f

This means the cache mode is well detected by the kernel right? So can't figure out why the cache does not work...

[SuperMaxusa]

This means the cache mode is well detected by the kernel right? So can't figure out why the cache does not work...

Most likely yes, the 9p cache modes was reworked about 2 years ago (https://github.com/torvalds/linux/commit/4eb3117888a923f6b9b1ad2dd093641c49a63ae5) and /proc/mounts now returns the hex value of the cache mode instead of the shortcut.