cloudstack
cloudstack copied to clipboard
apache
Cannot stop instance
ISSUE TYPE
- Bug Report
COMPONENT NAME
API(?), Storage
CLOUDSTACK VERSION
4.81.1
CONFIGURATION
OS / ENVIRONMENT
N/A
SUMMARY
sometimes an error occurs, the cause of which is not obvious and there is no indication where to look for it and where its source is. Just at one moment VMs refuse to shut down via Stop instance in the interface. When pressing the button, an error message appears:
(VM-0dde637a-eef3-4bb1-8f36-34a2a732ff0d) Unable to stop VM instance {‘id’:814, ‘instanceName’: ‘i-12-814-VM’, ‘type’: ‘User’, ‘uuid’: ‘0dde637a-eef3-4bb1-8f36-34a2a732ff0d’}
Log on agent:
2024-05-01 08:52:37,013 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:324bc3ba) Caught: java.lang.NullPointerException at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.lambda$getResourceByPath$0(LinstorStorageAdaptor.java:312) at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632) at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127) at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:528) at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.lambda$getResourceByPath$1(LinstorStorageAdaptor.java:312) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176) at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632) at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127) at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543) at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.getResourceByPath(LinstorStorageAdaptor.java:313) at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.disconnectPhysicalDiskByPath(LinstorStorageAdaptor.java:346) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDiskByPath(KVMStoragePoolManager.java:205) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3247) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:101) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:49) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1848) at com.cloud.agent.Agent.processRequest(Agent.java:662) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)
Log at management:
2024-05-01 10:29:32,915 DEBUG [c.c.a.t.Request] (AgentManager-Handler-4:null) (logid:) Seq 47-4875991021558694243: Processing: { Ans: , MgmtId: 20788631421694, via: 47, Ver: v1, Flags : 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"java.lang.NullPointerException at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.lambda$getResourceByPath$0(LinstorStorageAdaptor.java:312) at java.base/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90) at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632) at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127) at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230) at java.base/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:528) at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.lambda$getResourceByPath$1(LinstorStorageAdaptor.java:312) at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:176) at java.base/java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1632) at java.base/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127) at java.base/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:150) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:543) at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.getResourceByPath(LinstorStorageAdaptor.java:313) at com.cloud.hypervisor.kvm.storage.LinstorStorageAdaptor.disconnectPhysicalDiskByPath(LinstorStorageAdaptor.java:346) at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.disconnectPhysicalDiskByPath(KVMStoragePoolManager.java:205) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3247) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:101) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStopCommandWrapper.execute(LibvirtStopCommandWrapper.java:49) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1848) at com.cloud.agent.Agent.processRequest(Agent.java:662) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1082) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829) ","wait":"0","bypassHostMaintenance":"false"}}] }
The problem may not be in cloudstack itself, but I would be very grateful if someone could point me in which direction to look to find out the source of the problem.
I'm not sure what exact version you are really running, because the stacktrace doesn't match any of the 4.18. versions, but this is most probably already fixed with: https://github.com/apache/cloudstack/pull/8790
UPD: In fact, on the kvm server, the virtual instance stops (even though the UI shows an error and the status continues to be Running). But if immediately after the error try to shut down the VM with the Force switch on, the VM becomes Stopped. If this is not done, the VM will start again, apparently according to HA policy. I don't know if this information will help, but I hope so.
I'm not sure what exact version you are really running, because the stacktrace doesn't match any of the 4.18. versions, but this is most probably already fixed with: #8790
My version is 4.18.1.0
