cloudstack
cloudstack copied to clipboard
Published
20 hours ago •
apache
apache
Unable to list VMs on the remote KVM host, for import
Failed to list the VMs in remote KVM host, for importing them to CloudStack.
ISSUE TYPE
- Bug Report
COMPONENT NAME
VM Import
CLOUDSTACK VERSION
4.19.0
CONFIGURATION
KVM hypervisor / Advanced Zone
OS / ENVIRONMENT
Tried on MS, KVM Hosts on Ubuntu 22.04 & MS / KVM Hosts on Rocky Linux 8
SUMMARY
Failed to list the VMs in remote KVM host, for importing them to CloudStack.
API cmd has the following parameters - zoneid, host, username, password, hypervisor=kvm.
API Response:
{
"listvmsforimportresponse": {
"uuidList": [],
"errorcode": 530,
"cserrorcode": 4250,
"errortext": "Error while listing remote Vms"
}
}
Exception in Logs:
2024-02-26 06:12:25,881 ERROR [c.c.a.ApiServer] (qtp1753127384-473:ctx-f57f9de7 ctx-67aa987a) (logid:2887196b) unhandled exception executing api command: [Ljava.lang.String;@106067a6
com.cloud.utils.exception.CloudRuntimeException: Error while listing remote Vms
at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.getRemoteVms(UnmanagedVMsManagerImpl.java:2489)
at org.apache.cloudstack.vm.UnmanagedVMsManagerImpl.listVmsForImport(UnmanagedVMsManagerImpl.java:2464)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy459.listVmsForImport(Unknown Source)
at org.apache.cloudstack.api.command.admin.vm.ListVmsForImportCmd.execute(ListVmsForImportCmd.java:121)
STEPS TO REPRODUCE
- Go to Import-Export Instances in Tools section
- Select 'KVM' in Select Import-Export Source Hypervisor
- Pick option 'Import Instance from KVM Host'
- Add the remote host details, and Click on 'Fetch Instances'
EXPECTED RESULTS
The stopped VMs on the remote KVM host should be listed, for import
ACTUAL RESULTS
Unable to list VMs on the remote KVM, fails with Request Failed (530) error.
@sureshanaparti @kiranchavala I run into this issue last week. It seems to be because that the libvirt service on remote host listens on TLS port (16514), but the API call tries to get VMs on TCP port (16509). To access the TLS port, the cert/key are required. However they are not supported for now.
@sureshanaparti can you retry after the changes below ?
- listen_tls to 0 in libvirtd.conf
- listen_tcp to 1 in libvirtd.conf
- restart libvirtd
@weizhouapache Tried with listen_tcp = 1, but still unable to see list the stopped VMs.
with listen_tcp = 0, it's throwing exception and fails with Request Failed (530) error. I think, it's better to show the appropriate message if failed to connect to the remote host's qemu.
@weizhouapache
the issue is still present
Steps followed
- Create a KVM external environment
Launch a centos 7 vm and install the following packages
yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
Systemctl enable libvirtd
systemctl start libvirtd
- Edit the /etc/libvirtd/libvirtd.conf
listen_tls = 0
listen_tcp = 1
tcp_port = "16509"
auth_tcp = "none"
mdns_adv = 0
key_file="/etc/pki/libvirt/private/serverkey.pem"
cert_file="/etc/pki/libvirt/servercert.pem"
ca_file="/etc/pki/CA/cacert.pem"
- Edit the /etc/sysconfig/libvirtd
`LIBVIRTD_ARGS="--listen"' `
- Launch a vm with the following command
virt-install --virt-type=kvm --name centos8 --vcpus 2 --ram 2048 --os-variant=rhel8.0 --cdrom=/var/lib/libvirt/images/centos8.iso --network=default --disk size=16
- Shutdown the vm
virsh shutdown <name >
Agent Logs
2024-02-26 10:08:04,738 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:da44af16) Processing command: com.cloud.agent.api.GetRemoteVmsCommand
2024-02-26 10:08:04,739 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:da44af16) Looking for libvirtd connection at: qemu+tcp://10.0.32.71/system
2024-02-26 10:08:04,745 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) VM testkiran: powerstate = VIR_DOMAIN_SHUTOFF; vm state=PowerOff
2024-02-26 10:08:04,749 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Unable to retrieve unmanaged instance info.
java.lang.NullPointerException
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.getTagValue(LibvirtDomainXMLParser.java:431)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.extractCpuModeDef(LibvirtDomainXMLParser.java:518)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:390)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:90)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903)
at com.cloud.agent.Agent.processRequest(Agent.java:683)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-02-26 10:08:04,749 ERROR [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Error while fetching instance details
com.cloud.utils.exception.CloudRuntimeException: Unable to retrieve unmanaged instance info. null
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:110)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903)
at com.cloud.agent.Agent.processRequest(Agent.java:683)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2024-02-26 10:08:04,749 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Unable to retrieve unmanaged instance info. java.lang.NullPointerException at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.getTagValue(LibvirtDomainXMLParser.java:431)
@kiranchavala it looks the VMs have been listed successfully.
but the vm is not compatible with cloudstack. can you share the vm xml dump ?
virsh dumpxml --security-info <vm name>
@weizhouapache
virsh dumpxml --security-info testkiran
dumpxml
<domain type='kvm'>
<name>testkiran</name>
<uuid>aafaaabc-8657-4efc-9c52-3422d4e04088</uuid>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-model' check='partial'>
<model fallback='allow'/>
</cpu>
<clock offset='utc'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled='no'/>
<suspend-to-disk enabled='no'/>
</pm>
<devices>
<emulator>/usr/libexec/qemu-kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/ubuntu-22.04.qcow2'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</disk>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2'/>
<source file='/var/lib/libvirt/images/testkiran.qcow2'/>
<target dev='vdb' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</controller>
<interface type='network'>
<mac address='52:54:00:09:73:b8'/>
<source network='default'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='spicevmc'>
<target type='virtio' name='com.redhat.spice.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='1'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<graphics type='vnc' port='-1' autoport='yes'>
<listen type='address'/>
</graphics>
<graphics type='spice' autoport='yes'>
<listen type='address'/>
<image compression='off'/>
</graphics>
<sound model='ich6'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</sound>
<video>
<model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</video>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='2'/>
</redirdev>
<redirdev bus='usb' type='spicevmc'>
<address type='usb' bus='0' port='3'/>
</redirdev>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
</memballoon>
</devices>
</domain>
<model fallback='allow'/>
@kiranchavala the vm does not have a model specified. maybe that caused the issue. looks like a bug
2024-02-26 10:08:04,738 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:da44af16) Processing command: com.cloud.agent.api.GetRemoteVmsCommand
2024-02-26 10:08:04,739 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:da44af16) Looking for libvirtd connection at: qemu+tcp://10.0.32.71/system
2024-02-26 10:08:04,745 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) VM testkiran: powerstate = VIR_DOMAIN_SHUTOFF; vm state=PowerOff
2024-02-26 10:08:04,749 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Unable to retrieve unmanaged instance info.
java.lang.NullPointerException
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.getTagValue(LibvirtDomainXMLParser.java:431)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.extractCpuModeDef(LibvirtDomainXMLParser.java:518)
at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:390)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:90)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46)
at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903)
@kiranchavala I've created a PR for this issue #8720
@sureshanaparti are you going to work on this issue, or should we have @vishesh92 work on this? Thanks.
@weizhouapache
the issue is still present
Steps followed
- Create a KVM external environment
Launch a centos 7 vm and install the following packages
yum install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install Systemctl enable libvirtd systemctl start libvirtd
- Edit the /etc/libvirtd/libvirtd.conf
listen_tls = 0 listen_tcp = 1 tcp_port = "16509" auth_tcp = "none" mdns_adv = 0 key_file="/etc/pki/libvirt/private/serverkey.pem" cert_file="/etc/pki/libvirt/servercert.pem" ca_file="/etc/pki/CA/cacert.pem"
- Edit the /etc/sysconfig/libvirtd
`LIBVIRTD_ARGS="--listen"' `
- Launch a vm with the following command
virt-install --virt-type=kvm --name centos8 --vcpus 2 --ram 2048 --os-variant=rhel8.0 --cdrom=/var/lib/libvirt/images/centos8.iso --network=default --disk size=16
- Shutdown the vm
virsh shutdown <name >Agent Logs2024-02-26 10:08:04,738 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) (logid:da44af16) Processing command: com.cloud.agent.api.GetRemoteVmsCommand 2024-02-26 10:08:04,739 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-5:null) (logid:da44af16) Looking for libvirtd connection at: qemu+tcp://10.0.32.71/system 2024-02-26 10:08:04,745 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) VM testkiran: powerstate = VIR_DOMAIN_SHUTOFF; vm state=PowerOff 2024-02-26 10:08:04,749 DEBUG [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Unable to retrieve unmanaged instance info. java.lang.NullPointerException at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.getTagValue(LibvirtDomainXMLParser.java:431) at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.extractCpuModeDef(LibvirtDomainXMLParser.java:518) at com.cloud.hypervisor.kvm.resource.LibvirtDomainXMLParser.parseDomainXML(LibvirtDomainXMLParser.java:390) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:90) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903) at com.cloud.agent.Agent.processRequest(Agent.java:683) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) 2024-02-26 10:08:04,749 ERROR [resource.wrapper.LibvirtGetRemoteVmsCommandWrapper] (agentRequest-Handler-5:null) (logid:da44af16) Error while fetching instance details com.cloud.utils.exception.CloudRuntimeException: Unable to retrieve unmanaged instance info. null at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.getUnmanagedInstance(LibvirtGetRemoteVmsCommandWrapper.java:110) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:71) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtGetRemoteVmsCommandWrapper.execute(LibvirtGetRemoteVmsCommandWrapper.java:46) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1903) at com.cloud.agent.Agent.processRequest(Agent.java:683) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
@kiranchavala can you test with latest 4.19 or main ? the vm import issue should be fixed by #8720
Tested with latest 4.19, able to list the VMs (with listen_tcp = 1). Noticed inappropriate error messages when remote host is not reachable, etc - improved messages / logs in this PR #9204.
@weizhouapache Tried with listen_tcp = 1, but still unable to see list the stopped VMs.
with listen_tcp = 0, it's throwing exception and fails with Request Failed (530) error. I think, it's better to show the appropriate message if failed to connect to the remote host's qemu.
Appropriate messages, are addressed in PR https://github.com/apache/cloudstack/pull/9204.
