libvirt_domain failure on apply

[Jubblin]

error on running kubitect apply on ubuntu 22.04

Experimental features are subject to breaking changes in future minor or patch releases, based on feedback.

If you have feedback on the design of this feature, please open a GitHub issue to discuss it.

(and one more similar warning elsewhere)

Error: error creating libvirt domain: internal error: qemu unexpectedly closed the monitor: 2022-08-15T15:36:23.942067Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume': Permission denied

with module.host_localhost.module.worker_module["1"].libvirt_domain.vm_domain, on modules/vm/vm.tf line 71, in resource "libvirt_domain" "vm_domain": 71: resource "libvirt_domain" "vm_domain" {

Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2022-08-15T15:36:24.137145Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume': Permission denied

with module.host_localhost.module.master_module["1"].libvirt_domain.vm_domain, on modules/vm/vm.tf line 71, in resource "libvirt_domain" "vm_domain": 71: resource "libvirt_domain" "vm_domain" {

Error running Terraform apply: exit status 1

Error: error creating libvirt domain: internal error: qemu unexpectedly closed the monitor: 2022-08-15T15:36:23.942067Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume': Permission denied

with module.host_localhost.module.worker_module["1"].libvirt_domain.vm_domain, on modules/vm/vm.tf line 71, in resource "libvirt_domain" "vm_domain": 71: resource "libvirt_domain" "vm_domain" {

Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2022-08-15T15:36:24.137145Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume': Permission denied

with module.host_localhost.module.master_module["1"].libvirt_domain.vm_domain, on modules/vm/vm.tf line 71, in resource "libvirt_domain" "vm_domain": 71: resource "libvirt_domain" "vm_domain" {

[MusicDin]

Hi,

thank you for opening an issue.

The error states that access to /var/lib/libvirt/pools/local-k8s-cluster-main-resource-pool/base_volume is denied. If mainResourcePoolPath is not specified, Kubitect uses the path /var/lib/libvirt/pools/ for storing resource pools created during the apply. It appears that this path is not readable by the libvirt, resulting in a permission denied error.

If you set mainResourcePoolPath for a particular host to /var/lib/libvirt/images/ (this is libvirt's default path for storing ISO images), the problem should be solved.

hosts:
  - name: localhost 
    connection:
      type: local
    mainResourcePoolPath: /var/lib/libvirt/images/ # <--

To allow libvirt to use other directories, '/etc/libvirt/qemu.conf' should be modified accordingly.

Please let me know if this solves your problem or if you need further help.

[Jubblin]

I think I must be missing something here as I've made the change to mainResourcePoolPath as suggested but am now getting a near identical error.

Error: error creating libvirt domain: internal error: process exited while connecting to monitor: 2022-08-15T19:08:54.647873Z qemu-system-x86_64: -blockdev {"driver":"file","filename":"/var/lib/libvirt/images/kubitect-k8s-main-resource-pool/base_volume","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}: Could not open '/var/lib/libvirt/images/kubitect-k8s-main-resource-pool/base_volume': Permission denied

  with module.host_localhost.module.master_module["2"].libvirt_domain.vm_domain,
  on modules/vm/vm.tf line 71, in resource "libvirt_domain" "vm_domain":
  71: resource "libvirt_domain" "vm_domain" {

Adiitionally I've also done a ls so you can see the permissions.

:~/kubitect$ sudo ls -latrh /var/lib/libvirt/images/
total 32G
drwxr-xr-x 8 libvirt-qemu kvm  4.0K Aug 15 11:21 ..
drwx--x--x 3 libvirt-qemu kvm  4.0K Aug 15 20:08 .
drwx--x--x 2 root         root 4.0K Aug 15 20:08 kubitect-k8s-main-resource-pool
-rw------- 1 libvirt-qemu kvm   32G Aug 15 20:14 hassos_ova-5.13.qcow2

[MusicDin]

Before proceeding, just make sure that the user is added to the libvirt group? (kubitect should be used without root privileges)

1. List groups using groups $USER
2. If libvirt group is missing, add it using sudo usermode -aG libvirt $USER
3. Restart libvirt daemon service: sudo systemctl restart libvirtd.service

Also make sure that cluster resources are destroyed before reapplying the configuration.

kubitect destroy

---

Most likely either AppArmor or SELinux is preventing access to the /var/lib/libvirt/images.

One possible solution is to disable security_driver in /etc/libvirt/qemu.conf.

From AppArmor Wiki:

To disable the driver, either adjust /etc/libvirt/qemu.conf to have 'security_driver = “none”' or remove the AppArmor profile for libvirtd from the kernel and restart libvirtd.

# /etc/libvirt/qemu.conf
...
security_driver = "none"
...

Restart libvirt daemon service.

sudo systemctl restart libvirtd.service

Otherwise either AppArmor or SELinux needs to be configured.

Please let me know if this fixes you issue.

[MusicDin]

I am closing this issue as there has been no response for over 1 month.

If you need further help in solving this problem, feel free to reopen it.