QubesOS
Improve inter-VM file copy speed
Qubes OS version:
4
Affected component(s) or functionality:
qubes.Filecopy qvm-copy / qvm-move qvm-copy-to-vm / qvm-move-to-vm (dom0)
Steps to reproduce the behavior:
Assuming a just created disposable VM disp1234 in dom0:
time dd if=/dev/urandom of=~/test bs=1M count=1024
time qvm-copy-to-vm disp1234 ~/test
Expected or desired behavior:
It is reasonably fast / the second time should only be slightly above the first. At 50 MB/s I'd expect e.g. 20s.
Actual behavior:
It takes roughly ~57s on my machine (with SSD).
Interestingly, subsequent tries may cause it to reach ~20s even if you re-write the file content (i.e. make caching impossible). Apparently some target VM buffers are initially quite small and are enlarged upon subsequent attempts. If you switch to another disposable VM, you'll get the same slow copy times as before.
General notes:
qvm-copy between VMs of the same file takes ~50s on my system. I'm not sure why it appears to be slightly faster.
Other methods reliably work at ~20s, a comparison can be found at https://github.com/3hhh/blib/blob/master/lib/os/qubes4/dom0#L555
I have consulted the following relevant documentation:
.
I am aware of the following related, non-duplicate issues:
I didn't find any related bugs.
P.S.: Writing the 1G file in dom0 with dd takes roughly those 20s, i.e. that would be the max speed for my system it appears.
In dom0 cp 6sec, qvm-copy-to-vm 16s: not big enough difference that I've ever noticed.
Difference could be initial lvm thin pool allocation vs reuse of already allocated space after deletions.
Here's a full copy of my test run today:
> time dd if=/dev/urandom of=~/test bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 18.5109 s, 58.0 MB/s
real 0m18.626s
user 0m0.009s
sys 0m4.484s
> time qvm-copy-to-vm disp6293 ~/test
time qvm-copy-to-vm disp6293 ~/test
real 0m36.084s
user 0m7.868s
sys 0m7.132s
> qvm-run disp6293 'rm /home/user/QubesIncoming/dom0/test'
> time qvm-copy-to-vm disp6293 ~/test
real 0m32.209s
user 0m9.456s
sys 0m8.800s
> qvm-run disp6293 'rm /home/user/QubesIncoming/dom0/test'
> time qvm-copy-to-vm disp6293 ~/test
real 0m27.654s
user 0m6.248s
sys 0m5.066s
> qvm-shutdown disp6293
> time dd if=~/test bs=1M | qvm-run -p disp7242 'dd bs=1M of=/home/user/test'
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 20.5437 s, 52.3 MB/s
0+119087 records in
0+119087 records out
1073741824 bytes (1.1 GB, 1.0 GiB) copied, 20.2499 s, 53.0 MB/s
real 0m20.607s
user 0m3.821s
sys 0m9.039s
The interesting part here is that the qvm-run (qubes.Shell qrexec call or so?) variant gets close to the initial dd from urandom timings whereas the qvm-copy-to-vm variant (qubes.Filecopy) takes roughly twice as much time (looks like @unman can confirm that). Also, qvm-copy-to-vm appears to get better with subsequent tries.
@brendanhoar I'm on file pools, but unman probably isn't. So it's likely not related to pool driver implementations.
Side Note: Not sure why my machine is so much faster today in general. It's exactly the same hardware.
time dd if=~/test bs=1M | qvm-run -p disp7242 'dd bs=1M of=/home/user/test' 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 20.5437 s, 52.3 MB/s 0+119087 records in 0+119087 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 20.2499 s, 53.0 MB/s
real 0m20.607s user 0m3.821s sys 0m9.039s
The interesting part here is that the `qvm-run` (qubes.Shell qrexec call or so?) variant gets close to the initial `dd` from urandom timings whereas the `qvm-copy-to-vm` variant (qubes.Filecopy) takes roughly twice as much time (looks like @unman can confirm that). Also, `qvm-copy-to-vm` appears to get better with subsequent tries. @brendanhoar I'm on file pools, but unman probably isn't. So it's likely not related to pool driver implementations. Side Note: Not sure why my machine is so much faster today in general. It's exactly the same hardware.
Ah, it's possible it's a default block-size or similar issue on the sending or receiving sides.
I recall that default dd block size of 512 bytes was an issue on copying templates across lvm pools and when @marmarek increased the bs to something more reasonable (128KB?) on modern hardware, copying VMs between pools became substantially faster.
Something similar may be at play in the back end of qvm-copy and qvm-move. With your qvm-run piping, you're bypassing the back end of the qvm-copy and qvm-move programs and substituting your own dd with a better-optimized block size.
Just a guess, haven't looked at the code.
Brendan
PS - with the transition from HDD->SSD nearly complete, and now the transition from SATA to NVME in swing, it may be time to add a per-pool optimal block-size parameter to the data behind qvm-pool. Default to 128KB, allow user to increase it for NVME storage and/or decrease it for HDD storage. Utilize that value in operations coordinated from dom0 and make the value for each xen pv storage device (based on the pool it lives in) available to VM-side qubes code.
Looks like the copy buffer size is hard coded to 4K. From another thread, qrexec is now allowing for larger buffer sizes (from 4k to 64k?) perhaps in qubes 4.1 file_copy can increase buffer size to match? @marmarek ?
Looks like the copy buffer size is hard coded to 4K. From another thread, qrexec is now allowing for larger buffer sizes (from 4k to 64k?) perhaps in qubes 4.1 file_copy can increase buffer size to match?
Has there been any development about this?
