Increase or remove limit on number of block devices per qube (current limit: 26)

[3hhh]

How to file a helpful issue

Qubes OS release

4.1, 4.2

Brief summary

Attaching or creating more than 26 block devices to/in a VM fails.

The root cause appears to be a kernel limitation to handle any device higher than /dev/xvdz. It starts with /dev/xvda and ends with /dev/xvdz resulting in 26 block devices (/dev/xvda to /dev/xvdc are used by Qubes OS, so 23 remain for the user).

And yes, this tends to happen in practice with lots of loop devices.

Steps to reproduce

Attach more than 26 block devices to a VM or create loop devices inside a VM (I tested with debian-12 template).

Expected behavior

Works.

Actual behavior

Fails.

[rustybird]

It's possible to have way more than 26, but the frontend device currently has to be specified manually (tacking on more letters):

qvm-block attach -o frontend-dev=xvdabc ...

Without an explicit frontend-dev option it doesn't work because Qubes OS tooling only tries to find a free one up until xvdz: https://github.com/QubesOS/qubes-core-admin/blob/v4.3.0/qubes/ext/block.py#L44-L46

Not sure what's the exact limit - xvdzzz is too high and xen-blkfront complains about not being able to fit it into the block device's minor number.

[3hhh]

Uh. And I had considered this one to be tough nut to crack...

Thanks a lot for the hint, it's much appreciated!

[andrewdavidwong]

Is there any action left that should be taken on this issue?

[rustybird]

Just making it work without an explicit frontend-dev option.

[alimirjamali]

Just making it work without an explicit frontend-dev option.

This would add 676 new possibilities but I find this solution ugly:

AVAILABLE_FRONTENDS =  AVAILABLE_FRONTENDS + ['xvd'+c+d for c in string.ascii_lowercase for d in string.ascii_lowercase]

I emphasize Marek's note:

# TODO: get this from libvirt driver?

@rustybird I have been looking at libvirt's Python API and other documentation. I did not manage to find how to query backend Xen (or QEMU) limit of this. Had you any success?

[rustybird]

There might not be a limit that can be queried from libvirt (at least not an accurate one), as it will happily attempt to use some already out-of-range names like xvdzzz and even xvdzzzz.

This would add 676 new possibilities but I find this solution ugly:

I just noticed there's also another AVAILABLE_FRONTENDS definition in qubes.storage.Storage: https://github.com/QubesOS/qubes-core-admin/blob/v4.3.0/qubes/storage/init.py#L497

[alimirjamali]

Just found this in the Redhat docs:

A limit of 254 para-virtualized block devices per host exists. The total number of block devices attached to guests cannot exceed 254.

Also this note from SUSE docs:

When a virtual machine is running, each of its file-backed virtual disks consumes a loopback device on the host. By default, the host allows up to 64 loopback devices to be consumed.

I have not found the total limit per all guests.

[3hhh]

I tried to patch the aforementioned two files, but unfortunately attaching block devices still fails at /dev/xvde, i.e. xvde to xvdh are never created.

So the current limit appears to be 22.

[alimirjamali]

I just tested a patch. I successfully managed to attach 48 block devices from dom0 to a PVH qube. Here is the patch:

Initial patch to increase number of block devices per qube

diff --git a/qubes/ext/block.py b/qubes/ext/block.py
index 3a3eb1e5..46f80240 100644
--- a/qubes/ext/block.py
+++ b/qubes/ext/block.py
@@ -44,6 +44,7 @@ mode_re = re.compile(r"\A[rw]\Z")
 # TODO: get this from libvirt driver?
 AVAILABLE_FRONTENDS = ['xvd'+c for c in
                        string.ascii_lowercase[8:]+string.ascii_lowercase[:8]]
+AVAILABLE_FRONTENDS += ['xvda'+c for c in string.ascii_lowercase]
 
 SYSTEM_DISKS = ('xvda', 'xvdb', 'xvdc')
 # xvdd is considered system disk only if vm.kernel is set
diff --git a/qubes/storage/__init__.py b/qubes/storage/__init__.py
index b1464ab6..3a8e5494 100644
--- a/qubes/storage/__init__.py
+++ b/qubes/storage/__init__.py
@@ -495,6 +495,7 @@ class Storage:
     '''
 
     AVAILABLE_FRONTENDS = {'xvd' + c for c in string.ascii_lowercase}
+    AVAILABLE_FRONTENDS = AVAILABLE_FRONTENDS.union({'xvda' + c for c in string.ascii_lowercase})
 
     def __init__(self, vm):
         #: Domain for which we manage storage

And here is the script to automate creation of 48 virtual block devices and attaching them to untrusted qube:

Block device creation and attaching automation

#!/bin/sh

for i in {1..48}
do
[ ! -f block${i}.bin ] && dd if=/dev/zero of=block${i}.bin bs=512 count=10
sudo losetup --show /dev/loop100${i} block${i}.bin
qvm-block --verbose attach untrusted dom0:loop100${i} 
sleep 1
done

I believe that we are on the correct path. We have to increase the number and improve the patch until we break something. Then we could find out the exact number of possible devices. I will continue the work tomorrow.

[3hhh]

Qubes OS treats /dev/xvda to xvdh in a special way: Apparently those devices are hidden from tools such as qvm-block ls, which caused issues with my test code. Not sure whether that's a bug of its own. That's however likely why they are added to the list of available devices in the end.

Anyway I submitted a PR, which works for me. Feel free to review or comment on it there @alimirjamali. Thanks for your support!