cloudstack com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:

com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:

Open bkrajendra opened this issue 1 year ago • 15 comments

ISSUE TYPE

 * Bug Report

COMPONENT NAME

ADD HOST

CLOUDSTACK VERSION

4.17.0

CONFIGURATION

basic network

OS / ENVIRONMENT

Ubuntu 20.04

SUMMARY

I already had the host set up with an earlier version of the cloud stack. I had some issues with that setup, hence I decided to create new setup. I installed a fresh management server 4.17 on separate server. When I am trying add the host, I am getting following error.

2022-09-08 11:59:48,142 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2022-09-08 11:59:49,778 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
cat: /etc/cloudstack/agent/cloud.csr: No such file or directory
chmod: cannot access '/etc/cloudstack/agent/cloud.csr': No such file or directory

2022-09-08 11:59:49,778 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2022-09-08 11:59:50,381 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-488a4f32) (logid:f4dc8d86) HA health check task is running...
2022-09-08 11:59:51,437 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
cat: /etc/cloudstack/agent/cloud.csr: No such file or directory
chmod: cannot access '/etc/cloudstack/agent/cloud.csr': No such file or directory

2022-09-08 11:59:51,537 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2022-09-08 11:59:53,153 DEBUG [c.c.u.s.SSHCmdHelper] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
cat: /etc/cloudstack/agent/cloud.csr: No such file or directory
chmod: cannot access '/etc/cloudstack/agent/cloud.csr': No such file or directory

2022-09-08 11:59:53,154 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup keystore on the KVM host: IP
2022-09-08 11:59:53,156 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup keystore on the KVM host: IP
com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
2022-09-08 11:59:53,158 DEBUG [c.c.r.ResourceManagerImpl] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Could not add host at [http://IP] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup
keystore on the KVM host: IP].
com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP- Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:375)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        ... 60 more
2022-09-08 11:59:53,160 INFO  [c.c.u.e.CSExceptionErrorCode] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2022-09-08 11:59:53,161 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Exception:
com.cloud.exception.DiscoveryException: Could not add host at [http://IP] with zone [1], pod [1] and cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP- Failed to setup keystore on the KVM host: IP].
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:818)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy196.discoverHosts(Unknown Source)
        at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:142)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:375)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:644)
        ... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
        at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:320)
        at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:811)
        ... 60 more
2022-09-08 11:59:53,164 INFO  [c.c.a.ApiServer] (qtp555273695-21:ctx-661e3e65 ctx-dec74d3e) (logid:21946c0b) Could not add host at [http://IP] with zone [1], pod [1] and
cluster [1] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: IP - Failed to setup keystore on the KVM host: IP].

Sep 08 '22 12:09 bkrajendra

Thanks for opening your first issue here! Be sure to follow the issue template!

Sep 08 '22 12:09 boring-cyborg[bot]

@bkrajendra have you allowed ssh server on the KVM host to login as root user and password (password based auth)? You can also disable ca framework all together by setting the auth strictness global setting to false and try adding the host again (or restart the cloudstack-agent if it was setup already).

Sep 08 '22 12:09 rohityadavcloud

@rohityadavcloud thank you for the quick reply. Yes, I have access to the Host from the management server. I have tried access from the terminal and it works with password-based auth.

I'll try auth strictness global setting and test again and update here.

Sep 08 '22 16:09 bkrajendra

@rohityadavcloud I tried the following things:

set ca.plugin.root.auth.strictness=false restarted management server, restarted agent - still same error
uninstalled agent on a host, removed /etc/cloudstack/agent folder - now I got properties not found error and after that again same error related to failed to setup Keystore. Should I clean and install the host and try again?

Sep 09 '22 03:09 bkrajendra

@bkrajendra I am correct that your user for adding the host has this config in sudoers? cloudstack ALL=NOPASSWD: /usr/bin/cloudstack-setup-agent

Just out of curiosity: What's happening when you connect via ssh to the host while using the 'cloudstack' user and run sudo /usr/share/cloudstack-common/scripts/util/keystore-setup

Sep 09 '22 08:09 Hudratronium

@Hudratronium After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

About what you said: (in case of Ubuntu 20.04)

I am using the root user for all operations. I don't have cloudstack user. root ALL=(ALL) ALL Second, when I try keystore-setup from ssh, this is what I get:

sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
cat: : No such file or directory
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.

Sep 10 '22 03:09 bkrajendra

@Hudratronium After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

About what you said: (in case of Ubuntu 20.04)

I am using the root user for all operations. I don't have cloudstack user. root ALL=(ALL) ALL Second, when I try keystore-setup from ssh, this is what I get:
sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
cat: : No such file or directory
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.

same issue for me, no matter if via cloudstack, ssh or local

Sep 14 '22 13:09 Lueton

@Hudratronium After trying a lot and everything I gave up, and decided to go with an officially supported distro - Centos 7x minimal Host adding worked out of box without any issue, I am stuck with systemVMs are in starting state from last 12 hours, not sure how much time I should wait.

About what you said: (in case of Ubuntu 20.04)

I am using the root user for all operations. I don't have cloudstack user. root ALL=(ALL) ALL Second, when I try keystore-setup from ssh, this is what I get:
sudo /usr/share/cloudstack-common/scripts/util/keystore-setup
cat: : No such file or directory
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.
chmod: missing operand after ‘600’
Try 'chmod --help' for more information.

These errors when manually running keystore-setup are likely due to not providing all of the extra flags necessary to run keystore-setup script. They point to empty variables. You can possibly try something like this to see if it works on the Ubuntu system, or perhaps exposes a missing utility or path:

/usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/cloud.jks mykeystorepassword 365 /etc/cloudstack/agent/cloud.csr

As far as systemvm issues, that's separate debugging. Could be that the system image is not provisioned or it is up but the internal agent isn't reachable. Would need to find out if the VM is running and get on console, or if it isn't running find at what step it failed.

It may not strictly be necessary to use system VMs if you opt for "direct download" VM templates and don't need VNC console, but it's worth trying to find out what is wrong.

Sep 14 '22 15:09 mlsorensen

@bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

In case you still hit issues, let me know. I'm happy to update/edit the QIG and notes on my blog. cc @wido @weizhouapache any pointers if you can see, as you're one of the major Ubuntu users.

Sep 14 '22 16:09 rohityadavcloud

@bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

In case you still hit issues, let me know. I'm happy to update/edit the QIG and notes on my blog. cc @wido @weizhouapache any pointers if you can see, as you're one of the major Ubuntu users.

I installed the general Ubuntu Server 20.04. I'll try your installation guide as soon as possible, thanks for sharing!

Sep 14 '22 16:09 Lueton

@bkrajendra @Hudratronium @Lueton was Ubuntu minimal server installed? We don't support that, we want the standard Ubuntu server installed, where some of the utilities are preinstalled. (you can refer to my install notes on Ubuntu 20.04 here which works for me https://rohityadav.cloud/blog/cloudstack-kvm/)

We were using Ubuntu Server 20.04.0 - which should be completely supported as far as i am aware.

Sep 14 '22 21:09 Hudratronium

@Hudratronium yes, ubuntu 20.04 is offically fully supported in cloudstack.

Questions

was the host added in other cloudstack platform before ?
Did you clean the host before you add it to the new platform ?

Sep 15 '22 11:09 weizhouapache

I gave it another try with @rohityadavcloud guide. It still doesnt work yet but i discovered that the error message com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host only occours if i use a non-root user when adding a new host to the cluster. If i use root this message will not be in the logs.

However this might not be the main error. I can also see in the logs that theres another error (see screeenshot) which occours with both root and non-root user. I could see that the management node can establish a connection to the host (verified via ssh logs, host CPU increased) but then there is a problem (i guess with libvirt?) which causes to agent on the host to stop/restart.

Side note: the ui only responds to this when using non-root user. If i use root the ui responds only until i close the add host dialog.

Sep 16 '22 08:09 Lueton

I gave it another try with @rohityadavcloud guide. It still doesnt work yet but i discovered that the error message com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host only occours if i use a non-root user when adding a new host to the cluster. If i use root this message will not be in the logs.

However this might not be the main error. I can also see in the logs that theres another error (see screeenshot) which occours with both root and non-root user. I could see that the management node can establish a connection to the host (verified via ssh logs, host CPU increased) but then there is a problem (i guess with libvirt?) which causes to agent on the host to stop/restart.

Side note: the ui only responds to this when using non-root user. If i use root the ui responds only until i close the add host dialog.

@Lueton if you add the host using user/password, this error can be ignored.

Sep 16 '22 09:09 weizhouapache

@weizhouapache thats good to know, i was very confused!

this is my /agent/setup.log after trying to add this machine as a new host, maybe you can identify the issue.

root@delta:~# cat /var/log/cloudstack/agent/setup.log
DEBUG:root:execute:uname -r
DEBUG:root:execute:uname -m
DEBUG:root:execute:hostname -f
DEBUG:root:execute:kvm-ok
DEBUG:root:execute:awk '/MemTotal/ { printf "%.3f \n", $2/1024 }' /proc/meminfo
DEBUG:root:execute:ip a | grep "^\w" | grep -iv "^lo" | wc -l
DEBUG:root:execute:service apparmor status
DEBUG:root:execute:apparmor_status |grep libvirt
DEBUG:root:Failed to execute:
DEBUG:root:execute:/bin/systemctl mask libvirtd.socket
DEBUG:root:execute:/bin/systemctl mask libvirtd-ro.socket
DEBUG:root:execute:/bin/systemctl mask libvirtd-admin.socket
DEBUG:root:execute:/bin/systemctl mask libvirtd-tls.socket
DEBUG:root:execute:/bin/systemctl mask libvirtd-tcp.socket
DEBUG:root:execute:systemctl restart libvirtd
DEBUG:root:execute:ufw allow 22
DEBUG:root:execute:ufw allow 1798
DEBUG:root:execute:ufw allow 16509
DEBUG:root:execute:ufw allow 16514
DEBUG:root:execute:ufw allow proto tcp from any to any port 5900:6100
DEBUG:root:execute:ufw allow proto tcp from any to any port 49152:49216
DEBUG:root:execute:sudo /usr/sbin/service ufw status
DEBUG:root:execute:sudo /usr/sbin/service ufw stop
DEBUG:root:execute:sudo /usr/sbin/service ufw status
DEBUG:root:Failed to execute:● ufw.service - Uncomplicated firewall
     Loaded: loaded (/lib/systemd/system/ufw.service; enabled; vendor preset: enabled)
     Active: inactive (dead) since Fri 2022-09-16 10:17:25 UTC; 23ms ago
       Docs: man:ufw(8)
    Process: 2354 ExecStop=/lib/ufw/ufw-init stop (code=exited, status=0/SUCCESS)
   Main PID: 396 (code=exited, status=0/SUCCESS)

Sep 16 10:16:19 delta systemd[1]: Finished Uncomplicated firewall.
Sep 16 10:17:25 delta systemd[1]: Stopping Uncomplicated firewall...
Sep 16 10:17:25 delta systemd[1]: ufw.service: Succeeded.
Sep 16 10:17:25 delta systemd[1]: Stopped Uncomplicated firewall.
DEBUG:root:execute:sudo /usr/sbin/service ufw start
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent stop
DEBUG:root:execute:sleep 30
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent remove
DEBUG:root:execute:sudo update-rc.d -f cloudstack-agent defaults
DEBUG:root:Failed to execute:update-rc.d: error: unable to read /etc/init.d/cloudstack-agent
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent status
DEBUG:root:Failed to execute:● cloudstack-agent.service - CloudStack Agent
     Loaded: loaded (/lib/systemd/system/cloudstack-agent.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-09-16 10:17:28 UTC; 30s ago
       Docs: http://www.cloudstack.org/
   Main PID: 2216 (code=exited, status=143)

Sep 16 10:17:23 delta systemd[1]: Started CloudStack Agent.
Sep 16 10:17:23 delta java[2216]: SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
Sep 16 10:17:23 delta java[2216]: SLF4J: Defaulting to no-operation (NOP) logger implementation
Sep 16 10:17:23 delta java[2216]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Sep 16 10:17:27 delta systemd[1]: Stopping CloudStack Agent...
Sep 16 10:17:28 delta systemd[1]: cloudstack-agent.service: Main process exited, code=exited, status=143/n/a
Sep 16 10:17:28 delta systemd[1]: cloudstack-agent.service: Failed with result 'exit-code'.
Sep 16 10:17:28 delta systemd[1]: Stopped CloudStack Agent.
DEBUG:root:execute:sudo /usr/sbin/service cloudstack-agent start

Sep 16 '22 10:09 Lueton

Hi @Lueton could you have any progress on this error? Have checked the output of journalctl -xe?

Oct 31 '22 11:10 nvazquez

Hi @Lueton could you have any progress on this error? Have checked the output of journalctl -xe?

Hi! Unfortunately not. I have really tried everything I could think of. The only thing I noticed is that libvirt always crashes on the client and restarts when trying to connect it to the management host. But I could not get more information. I have also looked at all the logs that make sense to me. I'm not a Linux or cloud pro either so this may not be a bug but a lack of knowledge on my side.

Oct 31 '22 11:10 Lueton

@Lueton can you try to remove the host from CloudStack if it is on the database, retry adding it from the UI and share the management server logs?

Oct 31 '22 19:10 nvazquez

I gave it another try with @rohityadavcloud guide. It still doesnt work yet but i discovered that the error message com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host only occours if i use a non-root user when adding a new host to the cluster. If i use root this message will not be in the logs.

However this might not be the main error. I can also see in the logs that theres another error (see screeenshot) which occours with both root and non-root user. I could see that the management node can establish a connection to the host (verified via ssh logs, host CPU increased) but then there is a problem (i guess with libvirt?) which causes to agent on the host to stop/restart.

Side note: the ui only responds to this when using non-root user. If i use root the ui responds only until i close the add host dialog.

seems like i have same issue with this, i tried to uninstalling the agent and make sure all user can access the root but still cannot store the key, does the agent possibility are not compatible with the OS? i was using Ubuntu 22.04

Nov 29 '22 08:11 troke12

closed via #7090

Jan 18 '23 15:01 weizhouapache

I want to add a kvm host on cloudstack...while adding it using CS ui getting some error...any one have an idea ??

uierr

While checking cloudstack management server log...it showing

2024-04-12 08:55:42,249 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-360:ctx-627175fb ctx-825dd15b) (logid:b754aa80) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-04-12 08:55:43,301 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-360:ctx-627175fb ctx-825dd15b) (logid:b754aa80) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper
sudo: a password is required

2024-04-12 08:55:43,301 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-360:ctx-627175fb ctx-825dd15b) (logid:b754aa80)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.10.118 - Failed to setup keystore on the KVM host: 192.168.10.118
2024-04-12 08:55:43,301 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-360:ctx-627175fb ctx-825dd15b) (logid:b754aa80)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.10.118 - Failed to setup keystore on the KVM host: 192.168.10.118
com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host: 192.168.10.118
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:178)
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationIn

i tried to run the script getting below error...

sudo /usr/share/cloudstack-common/scripts/util/keystore-setup

Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying

Apr 12 '24 08:04 vishnuvs369

sudo: a terminal is required to read the password; either use the -S option to read from standard input or configure an askpass helper sudo: a password is required

@vishnuvs369 the error indicates that the user needs to be added to wheel group

Apr 12 '24 09:04 weizhouapache

@weizhouapache i have provided root as username and root user password for adding kvm host...now that error gone....now i'm getting different error.

2024-04-12 09:34:01,056 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying


2024-04-12 09:34:01,056 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-04-12 09:34:02,255 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-4:ctx-3f3b6330) (logid:f119a186) HA health check task is running...
2024-04-12 09:34:02,885 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-23a200d4) (logid:6cdcc245) AutoScaling Monitor is running...
2024-04-12 09:34:03,361 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
chmod: cannot access '/etc/cloudstack/agent/agent.properties': No such file or directory

2024-04-12 09:34:03,388 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
2024-04-12 09:34:04,431 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 09:34:04,431 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
2024-04-12 09:34:05,254 DEBUG [c.c.a.ApiServlet] (qtp239372207-359:ctx-d1046213) (logid:b74dfb77) ===START===  192.168.10.111 -- GET  command=readyForShutdown&response=json
2024-04-12 09:34:05,254 DEBUG [c.c.a.ApiServlet] (qtp239372207-359:ctx-d1046213) (logid:b74dfb77) Two factor authentication is already verified for the user 2, so skipping
2024-04-12 09:34:05,261 DEBUG [c.c.a.ApiServer] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-04-12 09:34:05,263 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) Account [Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]] is Root Admin or Domain Admin, all APIs are allowed.
2024-04-12 09:34:05,265 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [readyForShutdown] for user [User {"username":"admin","uuid":"8b9cdbeb-f746-11ee-9e73-000c29962f22"}.] as allowed.
2024-04-12 09:34:05,266 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-04-12 09:34:05,267 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-04-12 09:34:05,269 DEBUG [c.c.a.ApiServlet] (qtp239372207-359:ctx-d1046213 ctx-85baecad) (logid:b74dfb77) ===END===  192.168.10.111 -- GET  command=readyForShutdown&response=json
2024-04-12 09:34:05,497 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 09:34:05,497 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
2024-04-12 09:34:06,256 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-dcb4daff) (logid:1ac789dd) HA health check task is running...
2024-04-12 09:34:06,562 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties 9YXfhdTjZTpXOcSZ /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 09:34:06,562 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
2024-04-12 09:34:06,562 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiServer.queueCommand

Apr 12 '24 09:04 vishnuvs369

2024-04-12 09:34:01,056 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-305:ctx-ee051bc3 ctx-b69a7b97) (logid:3616be44) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/ SSH command output:Failed to generate CSR file, retrying after removing existing settings Reverting libvirtd to not listen on TLS Removing cloud.* files in /etc/cloudstack/agent Retrying to generate CSR file Failed to generate CSR file while retrying

@vishnuvs369 these certicicate errors are misleading please share the whole log on mgmt server and agent (agent.log, setup.log) when adding the host

Apr 12 '24 10:04 weizhouapache

@weizhouapache Below given full log of management server while adding an host..

tail -f /var/log/cloudstack/management/management-server.log

2024-04-12 11:48:39,663 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying


2024-04-12 11:48:39,663 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-04-12 11:48:41,860 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
chmod: cannot access '/etc/cloudstack/agent/agent.properties': No such file or directory

2024-04-12 11:48:41,886 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
2024-04-12 11:48:42,774 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-2784ff02) (logid:0eda3e49) HA health check task is running...
2024-04-12 11:48:42,788 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5) (logid:95a2f6aa) ===START===  192.168.10.111 -- GET  command=readyForShutdown&response=json
2024-04-12 11:48:42,788 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5) (logid:95a2f6aa) Two factor authentication is already verified for the user 2, so skipping
2024-04-12 11:48:42,794 DEBUG [c.c.a.ApiServer] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-04-12 11:48:42,797 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) Account [Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]] is Root Admin or Domain Admin, all APIs are allowed.
2024-04-12 11:48:42,798 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [readyForShutdown] for user [User {"username":"admin","uuid":"8b9cdbeb-f746-11ee-9e73-000c29962f22"}.] as allowed.
2024-04-12 11:48:42,799 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-04-12 11:48:42,800 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-04-12 11:48:42,802 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) ===END===  192.168.10.111 -- GET  command=readyForShutdown&response=json
2024-04-12 11:48:42,929 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 11:48:42,929 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
2024-04-12 11:48:43,414 DEBUG [c.c.v.VirtualMachineManagerImpl] (Vm-Operations-Cleanup-1:ctx-e1dfc0f1) (logid:c70e7745) VM Operation Thread Running
2024-04-12 11:48:43,420 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-6086b31e) (logid:e94208a8) ClusterDRS.poll is being called at 2024-04-12 11:49:00 GMT
2024-04-12 11:48:43,425 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-6086b31e) (logid:e94208a8) Removed 0 old drs migration plans
2024-04-12 11:48:43,431 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) VM scheduler.poll is being called at 2024-04-12 11:49:00 GMT
2024-04-12 11:48:43,434 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) Got 0 scheduled jobs to be executed at 2024-04-12 11:49:00 GMT
2024-04-12 11:48:43,436 INFO  [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) Cleaned up 0 VM scheduled job entries
2024-04-12 11:48:43,994 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 11:48:43,994 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
2024-04-12 11:48:45,058 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 11:48:45,058 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
2024-04-12 11:48:45,058 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
2024-04-12 11:48:45,059 DEBUG [c.c.r.ResourceManagerImpl] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Could not add host at [http://192.168.10.118] with zone [6], pod [7] and cluster [6] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	... 60 more
2024-04-12 11:48:45,060 INFO  [c.c.u.e.CSExceptionErrorCode] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2024-04-12 11:48:45,060 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Exception: 
com.cloud.exception.DiscoveryException: Could not add host at [http://192.168.10.118] with zone [6], pod [7] and cluster [6] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:834)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	... 60 more

i could'nt see any log for setup...while checking on the path /var/log/cloudstack/agent/ is an empty directory

Screenshot from 2024-04-12 17-23-45

Apr 12 '24 11:04 vishnuvs369

@weizhouapache Below given full log of management server while adding an host..

tail -f /var/log/cloudstack/management/management-server.log

2024-04-12 11:48:39,663 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:Failed to generate CSR file, retrying after removing existing settings
Reverting libvirtd to not listen on TLS
Removing cloud.* files in /etc/cloudstack/agent
Retrying to generate CSR file
Failed to generate CSR file while retrying


2024-04-12 11:48:39,663 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
2024-04-12 11:48:41,860 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-setup /etc/cloudstack/agent/agent.properties /etc/cloudstack/agent/
SSH command output:
chmod: cannot access '/etc/cloudstack/agent/agent.properties': No such file or directory

2024-04-12 11:48:41,886 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
2024-04-12 11:48:42,774 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-1:ctx-2784ff02) (logid:0eda3e49) HA health check task is running...
2024-04-12 11:48:42,788 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5) (logid:95a2f6aa) ===START===  192.168.10.111 -- GET  command=readyForShutdown&response=json
2024-04-12 11:48:42,788 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5) (logid:95a2f6aa) Two factor authentication is already verified for the user 2, so skipping
2024-04-12 11:48:42,794 DEBUG [c.c.a.ApiServer] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) CIDRs from which account 'Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]' is allowed to perform API calls: 0.0.0.0/0,::/0
2024-04-12 11:48:42,797 INFO  [o.a.c.a.DynamicRoleBasedAPIAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) Account [Account [{"accountName":"admin","id":2,"uuid":"8b9bbc51-f746-11ee-9e73-000c29962f22"}]] is Root Admin or Domain Admin, all APIs are allowed.
2024-04-12 11:48:42,798 WARN  [o.a.c.a.ProjectRoleBasedApiAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) Project is null, ProjectRoleBasedApiAccessChecker only applies to projects, returning API [readyForShutdown] for user [User {"username":"admin","uuid":"8b9cdbeb-f746-11ee-9e73-000c29962f22"}.] as allowed.
2024-04-12 11:48:42,799 DEBUG [o.a.c.a.StaticRoleBasedAPIAccessChecker] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) RoleService is enabled. We will use it instead of StaticRoleBasedAPIAccessChecker.
2024-04-12 11:48:42,800 DEBUG [o.a.c.r.ApiRateLimitServiceImpl] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) API rate limiting is disabled. We will not use ApiRateLimitService.
2024-04-12 11:48:42,802 DEBUG [c.c.a.ApiServlet] (qtp239372207-307:ctx-2d97f1c5 ctx-bd572d21) (logid:95a2f6aa) ===END===  192.168.10.111 -- GET  command=readyForShutdown&response=json
2024-04-12 11:48:42,929 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 11:48:42,929 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
2024-04-12 11:48:43,414 DEBUG [c.c.v.VirtualMachineManagerImpl] (Vm-Operations-Cleanup-1:ctx-e1dfc0f1) (logid:c70e7745) VM Operation Thread Running
2024-04-12 11:48:43,420 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-6086b31e) (logid:e94208a8) ClusterDRS.poll is being called at 2024-04-12 11:49:00 GMT
2024-04-12 11:48:43,425 DEBUG [o.a.c.c.ClusterDrsServiceImpl] (VMSchedulerPollTask:ctx-6086b31e) (logid:e94208a8) Removed 0 old drs migration plans
2024-04-12 11:48:43,431 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) VM scheduler.poll is being called at 2024-04-12 11:49:00 GMT
2024-04-12 11:48:43,434 DEBUG [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) Got 0 scheduled jobs to be executed at 2024-04-12 11:49:00 GMT
2024-04-12 11:48:43,436 INFO  [o.a.c.v.s.VMSchedulerImpl] (VMSchedulerPollTask:ctx-980e9c28) (logid:f9c3982d) Cleaned up 0 VM scheduled job entries
2024-04-12 11:48:43,994 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 11:48:43,994 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Executing cmd: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
2024-04-12 11:48:45,058 DEBUG [c.c.u.s.SSHCmdHelper] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) SSH command: sudo /usr/share/cloudstack-common/scripts/util/keystore-cert-import /etc/cloudstack/agent/agent.properties y5F5wnMdvsJSAFOw /etc/cloudstack/agent/
SSH command output:Failed to find keystore passphrase from file: /etc/cloudstack/agent/agent.properties, quitting!


2024-04-12 11:48:45,058 WARN  [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
2024-04-12 11:48:45,058 DEBUG [c.c.h.k.d.LibvirtServerDiscoverer] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620)  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
2024-04-12 11:48:45,059 DEBUG [c.c.r.ResourceManagerImpl] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Could not add host at [http://192.168.10.118] with zone [6], pod [7] and cluster [6] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	... 60 more
2024-04-12 11:48:45,060 INFO  [c.c.u.e.CSExceptionErrorCode] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Could not find exception: com.cloud.exception.DiscoveryException in error code list for exceptions
2024-04-12 11:48:45,060 WARN  [o.a.c.a.c.a.h.AddHostCmd] (qtp239372207-304:ctx-e8455b79 ctx-8e6fb9a4) (logid:c5b46620) Exception: 
com.cloud.exception.DiscoveryException: Could not add host at [http://192.168.10.118] with zone [6], pod [7] and cluster [6] due to: [ can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!].
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:834)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
	at com.sun.proxy.$Proxy195.discoverHosts(Unknown Source)
	at org.apache.cloudstack.api.command.admin.host.AddHostCmd.execute(AddHostCmd.java:136)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiServer.queueCommand(ApiServer.java:803)
	at com.cloud.api.ApiServer.handleRequest(ApiServer.java:624)
	at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:342)
	at com.cloud.api.ApiServlet$1.run(ApiServlet.java:149)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:146)
	at com.cloud.api.ApiServlet.doPost(ApiServlet.java:105)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
	at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
	at org.eclipse.jetty.server.Server.handle(Server.java:516)
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: com.cloud.exception.DiscoveredWithErrorException:  can't setup agent, due to com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually! - Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:376)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	at com.cloud.resource.ResourceManagerImpl.discoverHosts(ResourceManagerImpl.java:660)
	... 59 more
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to setup certificate in the KVM agent's keystore file, please see logs and configure manually!
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.setupAgentSecurity(LibvirtServerDiscoverer.java:206)
	at com.cloud.hypervisor.kvm.discoverer.LibvirtServerDiscoverer.find(LibvirtServerDiscoverer.java:321)
	at com.cloud.resource.ResourceManagerImpl.discoverHostsFull(ResourceManagerImpl.java:827)
	... 60 more

i could'nt see any log for setup...while checking on the path /var/log/cloudstack/agent/ is an empty directory

Screenshot from 2024-04-12 17-23-45

can you upload the logs right after command=addHost ?

Apr 12 '24 12:04 weizhouapache

cloudstack cloudstack copied to clipboard

com.cloud.utils.exception.CloudRuntimeException: Failed to setup keystore on the KVM host:

ISSUE TYPE

COMPONENT NAME

CLOUDSTACK VERSION

CONFIGURATION

OS / ENVIRONMENT

SUMMARY

About what you said: (in case of Ubuntu 20.04)

About what you said: (in case of Ubuntu 20.04)

About what you said: (in case of Ubuntu 20.04)

cloudstack
cloudstack copied to clipboard