lima fails in a confusing way when the home directory is on NFS

[DrDaveD]

Description

I stumbled across the fact that a home directory on NFS does not work with lima, because qemu can't handle it. I discussed it with @afbjorklund and he requested that I make an issue so something can be done to make it more clear why a failure is happening.

The error I got looked like this:

$ limactl start
? Creating an instance "default" Choose another template (docker, podman, archlinux, fedora, ...)
? Choose a template ubuntu-lts
? Creating an instance "default" Proceed with the current configuration
INFO[0014] Starting the instance "default" with VM driver "qemu"
INFO[0014] Attempting to download the image              arch=x86_64 digest="sha256:42dcf9757e75c3275486b397a752fb535c7cd8e5232ee5ee349554b7a55f1702" location="https://cloud-images.ubuntu.com/releases/22.04/release-20240308/ubuntu-22.04-server-cloudimg-amd64.img"
Downloading the image (ubuntu-22.04-server-cloudimg-amd64.img)
618.25 MiB / 618.25 MiB [----------------------------------] 100.00% 31.03 MiB/s
INFO[0038] Downloaded the image from "https://cloud-images.ubuntu.com/releases/22.04/release-20240308/ubuntu-22.04-server-cloudimg-amd64.img"
FATA[0038] failed to run [qemu-img create -f qcow2 -F qcow2 -b /nashome/d/dwd/.lima/default/basedisk /nashome/d/dwd/.lima/default/diffdisk 107374182400]: "Formatting '/nashome/d/dwd/.lima/default/diffdisk', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=107374182400 backing_file=/nashome/d/dwd/.lima/default/basedisk backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16\nqemu-img: /nashome/d/dwd/.lima/default/diffdisk: Image is not in qcow2 format\n": exit status 1

The workaround is to set LIMACTL_HOME to an empty directory on a local disk. If you point it to a directory that has other subdirectories in it, lima unexpectedly dies looking for lima.yaml files in them. I expected a variable name with HOME in it to look like $HOME where a .lima directory would be created; instead it behaves more like a CONFIG directory, so that name was also misleading.

[afbjorklund]

The issue is https://gitlab.com/qemu-project/qemu/-/issues/946, about file locks being broken on NFS (known fact)

The suggestion is to document the workaround, and to add a check on lima home and warn if it is mounted as nfs

It is OK to have $HOME be on a network disk, including ~/.cache.

We just want $LIMA_HOME to be a on a local disk, for performance.

[afbjorklund]

The current error messages are a bit blunt, if you try to run in an unsupported fashion:

# limactl ls
FATA[0000] must not run as the root

$ limactl ls
FATA[0000] must not run on NFS dir

Unfortunately it is up to the administrator, to configure $LIMA_HOME on a local disk...

e.g. if "/local" is the mount point used:

export LIMA_HOME=/local/$USER/.lima