cannot boot VM with `writableStoreOverlay` + shares

[kraftnix]

I am unable to use a writable store when using shares, the documentation states that the path at microvm.writableStoreOverlay must be writable, and #20 mentions that this should work with a writabe overlay on share.

I can get a writable store using microvm.volumes, i.e.

{ config, ... }:
{
  microvm.writableStoreOverlay = "/nix/.rw-store";
  microvm.volumes = [{
    mountPoint = config.microvm.writableStoreOverlay;
    image = "store.img";
    size = 2000;
  }];
}

however trying to use a share instead with something like:

{ config, ... }:
{
  microvm.writableStoreOverlay = "/nix/.rw-store";
  microvm.shares = [
    {
      source = "/tmp/test-writable-store";
      mountPoint = config.microvm.writableStoreOverlay;
      tag = "rw-store";
      proto = "virtiofs";
      socket = "nix-store-rw.sock";
    }
  ];
}

results in a VM which does not boot, stating that the upper fs is missing required features.

mounting overlay filesystem on /nix/store...
[    0.324712] overlayfs: upper fs missing required features.
mount: mounting overlay on /mnt-root/nix/store failed: Invalid argument

This was tested with cloud-hypervisor specifically.

I couldn't find a test or example which shows this usecase, is this meant to be supported or am I doing something wrong?

[astro]

Apologies, I only tested this feature with volumes. Do they work for you?

Alternatively, is there anything we could do about it in our custom stage-1?

I will update the docs to reflect the current situation.

[kraftnix]

Yes, volumes work fine with the writableStoreOverlay. I am not 100% sure whether it will be possible to support the writable overlay on virtiofsd shares, I ran into similar errors when I was attempting to run docker with it's overlayfs image driver on top of a virtiofsd share within an nspawn container before, I never found a fix except just not using virtiofsd, and instead sharing a block device/img (as done here with volumes).

I might have another go at it later, but for now volumes does support my use case, thanks for updating the docs to reflect that!

note: I may be misremembering not finding a fix for the situation, I believe I have a stale branch somewhere that passed in custom options to virtiofsd-rs that may allow overlayfs to work.

[kraftnix]

So I looked at my old notes where I encountered the issue before and did some testing based on other similar/reported issues with overlayfs on virtiofs, however I couldn't get overlayfs to mount the writable store overlay even after trying different mount options within stage-1.

Some suggestions pointed to using fuse-overlayfs instead, and it managed to mount (with no errors), however there seems to be an error somewhere in stage-2, since the store seems to be mounted (I ran a cat on the init script successfully).

mounting overlay filesystem on /nix/store...
cat /mnt-root//nix/store/gq2ljfy7gckln3mi4czi2s38x2bl825c-nixos-system-dev-test-22.11.20220609.46b9f39/init
<... removed ...>
stage 2 init script (/mnt-root//nix/store/gq2ljfy7gckln3mi4czi2s38x2bl825c-nixos-system-dev-test-22.11.20220609.46b9f39/init) not found

I have tried with quite a few combinations of virtiofs, overlayfs and fuse-overlayfs and fuse-overlayfs seems to get closest (at a great performance cost) and I couldn't quite get it working.

Thinking about it I cannot actually see a use case where we would need a full writable share instead of a volume, so if it is too complex it may be not be worth supporting, at least volumes will fit my use case for development machines.