Backup failed with permission denied; now snapshot fails

[AdrianVollmer]

First: great project, thanks for publishing it! Now my issue:

I tried to backup a live VM and it failed with this error:

libvirt.libvirtError: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied

Then I manually deleted the snapshot image. Now it fails with this:

libvirt: error code 1: internal error: unable to execute QEMU command 'transaction': Error: Trying to create an image with the same filename as the backing file 
Failed to create domain snapshot

I was root the entire time. Not sure what to do now.

[milkey-mouse]

Was it the snapshot image or the backing image that QEMU could not open in the first error? Can you run stat on the file?

When you manually deleted the snapshot image, did you make sure qemu knew about it? In other words, does it show up when you run virsh domblklist <domain>?

[AdrianVollmer]

To your first question: I don't know, all it gave me was the error message above.

Unfortunately I cleaned it up and in the process made the VM unbootable, so I can't do any diagnostics anymore at this point.

I did not make sure.

[milkey-mouse]

I cleaned it up and in the process made the VM unbootable

I'm really sorry about that! Did you delete the overlay disk image (the one it tried to block-commit) or did you merge it in manually?

What backup-vm should do is attempt a libvirt/qemu block commit if the VM is running while being backed up:

https://github.com/milkey-mouse/backup-vm/blob/442ce385ca66fb254105ce566dfcdf3fe36da7b8/backup_vm/snapshot.py#L78-L81

And if the VM is turned off it should just run qemu-img commit like a human would:

https://github.com/milkey-mouse/backup-vm/blob/442ce385ca66fb254105ce566dfcdf3fe36da7b8/backup_vm/snapshot.py#L127

It's weird that there's a permission error not only because you're root but because backup-vm isn't the one trying to create/read that snapshot file. If the VM is running (which it seems like it was in your case), the only thing that touches the images is libvirt/QEMU using the permissions of the daemon (normally the libvirt-qemu user I think?) It seems like libvirt was able to create the file and QEMU was able to open it (for writing) the first time (when it pivots to the snapshot at the beginning of the backup) but couldn't open it for reading later for some reason...

For future reference, if a similar issue comes up, it would be safer to run virsh destroy <domain> to immediately power off and run qemu-img commit /var/lib/libvirt/images/<domain>-<disk>-tempsnap.qcow2 like backup-vm would have. This is what I did during early development when restores didn't work yet.

[AdrianVollmer]

I just tried again, here is the full error message. I could stat both files. Thanks for the hint in the last paragraph. That worked, but obviously my goal is to be able to back up a VM without having to shut it down....

$ backup-vm Windows10 /mnt/external/backups/borg2::vm-test
backup progress: 100%
libvirt: error code 1: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied
Traceback (most recent call last):
  File "/usr/local/bin/backup-vm", line 11, in <module>
    load_entry_point('backup-vm==0.1.dev28+g442ce38', 'console_scripts', 'backup-vm')()
  File "/usr/local/lib/python3.6/dist-packages/backup_vm-0.1.dev28+g442ce38-py3.6.egg/backup_vm/backup.py", line 54, in main
    borg_failed = multi.assimilate(args.archives)
  File "/usr/local/lib/python3.6/dist-packages/backup_vm-0.1.dev28+g442ce38-py3.6.egg/backup_vm/snapshot.py", line 175, in __exit__
    self.blockcommit(disks_to_backup)
  File "/usr/local/lib/python3.6/dist-packages/backup_vm-0.1.dev28+g442ce38-py3.6.egg/backup_vm/snapshot.py", line 81, in blockcommit
    | libvirt.VIR_DOMAIN_BLOCK_COMMIT_SHALLOW) < 0:
  File "/usr/lib/python3/dist-packages/libvirt.py", line 708, in blockCommit
    if ret == -1: raise libvirtError ('virDomainBlockCommit() failed', dom=self)
libvirt.libvirtError: internal error: unable to execute QEMU command 'block-commit': Could not reopen file: Permission denied

[tim-seoss]

FYI, I'm also seeing this running on a Debian server (in both -stable (stretch) and -testing (buster)) - libvirt versions 3.0.0 and 4.3.0 respectively.

The failure occurs during the call to self.dom.snapshotCreateXML(snapshot_xml, snapshot_flags).

It looks like libvirtd is asking qemu to create the snapshot .qcow2 file but this is failing.

A call to:

virsh snapshot-create --domain testdomain --disk-only --no-metadata --atomic

works as expected, but if I change the code to dump out the xml and then I pass that like this:

virsh snapshot-create --domain testdomain --disk-only --no-metadata --atomic --xmlfile snapshot_test.xml

Then the result is:

error: internal error: unable to execute QEMU command 'transaction': Could not create file: Permission denied

I checked the xml vs.

https://libvirt.org/formatsnapshot.html

and it looks sane to me.

Trying chmod 777 /path/to/libvirt-store-dir doesn't fix it, so it's definitely not a file permissions problem.

@AdrianVollmer - what OS and/or libvirt versions are you using?

[AdrianVollmer]

$ libvirtd --version
libvirtd (libvirt) 4.0.0
$ lsb_release -a
No LSB modules are available.
Distributor ID: Debian
Description:    Debian GNU/Linux unstable (sid)
Release:        unstable
Codename:       sid

[tim-seoss]

I think the problem might be that backup-vm is trying to create a snapshot of an IDE CDROM device which is pointing at an .iso

If I remove that device from the domain definition, then the backup works...

[Aleksei-Badyaev]

Same issue for me after upgrade from Debian 9 stretch to Debian 10 buster. My virtual domains do not have CDROM devices. I think issue is in new version of libvirt in new Debian distro - 10.0 buster.

[pauvos]

@Aleksei-Badyaev , seems to be an apparmor issue with debian 10. This (temporary) solution allows me to create snapshots again... but "virsh blockcommit" still doesn't work: https://askubuntu.com/a/491304

[ddormer]

There's a thread on the libvirt-users list about this problem but without a resolution at this time: https://www.redhat.com/archives/libvirt-users/2019-May/msg00064.html

[HristoKolev]

What worked for me was editing /etc/libvirt/qemu.conf and setting user = "root" and rebooting. Obviously it's a security issue, be sure to revert it back when the issue is fixed.

[HristoKolev]

I found out what caused the problem for me. It was the seclabel tag in the vm definition. Removing it does not solve the problem. Replacing it with this <seclabel type='none'/> does.

[jersobh]

I don't have seclabel but I have the same issue.

[HristoKolev]

Can you give us the VM definition? virsh dumpxml vmname

[migae21]

I doesnet have the seclevel in th VM definitions, but i have this in /ect/libvirt/qemu.conf. If i change it to: security_driver = "none" All is fine with commiting snapshots

[jersobh]

I've found the same as @migae21 .

[Jackfritt]

Same problem here I couldn´t blockcommit. As a temporary workaround I did sudo aa-teardown

then the Blockcommit worked like a charm. After that you should enable apparmor profiles again with sudo service apparmor restart and check status sudo aa-status

There is a bugreport https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=932456

[fbett]

Very funny, i also ran into this problem and gave up.

I discovered accidentally that apparmor will deny any access to disk images, whose filename does not end with an image file extension (.qcow2 in my case).

So today i tried again to create and commit snapshots without stopping apparmor.

And it's amazing: It works!

Both disk images (the original disk image and the intermediate snapshot disk image) need an image file extension (.qcow2 in my case). To force that the intermediate disk image file path has this file extension, you have to define the file path yourself, while creating the snapshot.

Example:

virsh snapshot-create-as --domain DOMAIN SNAPSHOT_NAME \
  --diskspec sda,file=/path/file-snapshot.qcow2 \
  --disk-only --atomic --quiesce --no-metadata

[milkey-mouse]

I found the AppArmor profile responsible. Its only method of whitelisting image files is by their extension...

However I can't see why this would be an issue. backup-vm always creates its snapshots as .qcow2:

https://github.com/milkey-mouse/backup-vm/blob/d7b69b1bf43f1859775017fbc9fd0a9abf0c458c/backup_vm/backup.py#L36

[milkey-mouse]

A gross workaround for this (if AppArmor works how I think it does...) would be to symlink/hardlink to the original disk under a .qcow2 or .img filename if the snapshot (i.e. virDomainBlockCommit()) fails.