cloudstack icon indicating copy to clipboard operation
cloudstack copied to clipboard

Published 20 hours ago • verified publisher icon apache

Cannot add CEPH as a primary storage on RH8-based linux distributions

Open vladimirpetrov opened this issue 2 years ago • 3 comments

ISSUE TYPE
  • Bug Report
COMPONENT NAME
API, Storage
CLOUDSTACK VERSION
4.17, 4.16
CONFIGURATION

CEPH Storage Vesion 15.2.16 (d46a73d6d0a67a79558054a3a5a72cb561724974) octopus (stable)

OS / ENVIRONMENT

Rocky linux 8, AlmaLinux 8 (as both management and host OS)

SUMMARY

When you try to add a CEPH storage on a RedHat 8 based linux distribution, the following error occurs:

Request failed. (530) Failed to add data store: Failed to access storage pool

Note: The same operation is executed successfully on CentOS 7 based environment.

STEPS TO REPRODUCE
Try to add CEPH as primary storage.
EXPECTED RESULTS
The CEPH storage is successfully added.
ACTUAL RESULTS
2022-06-16 08:29:27,917 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:null) (logid:) Seq 1-3410069342849597487: Processing:  { Ans: , MgmtId: 167781118, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a3462cc3-ba5d-3393-be89-fd7dabc05fe6
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }
2022-06-16 08:29:27,917 DEBUG [c.c.a.t.Request] (qtp681094281-21:ctx-d800c708 ctx-3242cdf1) (logid:812ac3c4) Seq 1-3410069342849597487: Received:  { Ans: , MgmtId: 167781118, via: 1(ref-trl-3289-k-Mr8-vladimir-petrov-kvm1), Ver: v1, Flags: 10, { Answer } }
2022-06-16 08:29:27,917 DEBUG [c.c.a.m.AgentManagerImpl] (qtp681094281-21:ctx-d800c708 ctx-3242cdf1) (logid:812ac3c4) Details from executing class com.cloud.agent.api.ModifyStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a3462cc3-ba5d-3393-be89-fd7dabc05fe6
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
(...)
2022-06-16 08:29:27,924 WARN  [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp681094281-21:ctx-d800c708 ctx-3242cdf1) (logid:812ac3c4) Unable to establish a connection between Host {"id": "1", "
name": "ref-trl-3289-k-Mr8-vladimir-petrov-kvm1", "uuid": "b2a4276c-8413-4ef1-84ed-7fb5276f1dad", "type"="Routing"} and org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@1e171ac
com.cloud.utils.exception.CloudRuntimeException: Unable establish connection from storage head to storage pool 3 due to com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a346
2cc3-ba5d-3393-be89-fd7dabc05fe6
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
3
        at org.apache.cloudstack.storage.datastore.provider.DefaultHostListener.hostConnect(DefaultHostListener.java:134)
        at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1110)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy66.connectHostToSharedPool(Unknown Source)
        at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.attachCluster(CloudStackPrimaryDataStoreLifeCycleImpl.java:453)
        at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:823)
        at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:237)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy66.createPool(Unknown Source)
        at org.apache.cloudstack.api.command.admin.storage.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:163)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:163)
        at com.cloud.api.ApiServer.queueCommand(ApiServer.java:776)
        at com.cloud.api.ApiServer.handleRequest(ApiServer.java:600)
        at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:327)
        at com.cloud.api.ApiServlet$1.run(ApiServlet.java:145)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:142)
        at com.cloud.api.ApiServlet.doPost(ApiServlet.java:101)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:665)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
        at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:550)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1434)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1349)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
        at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:763)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
        at org.eclipse.jetty.server.Server.handle(Server.java:516)
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:400)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:645)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:392)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
        at java.base/java.lang.Thread.run(Thread.java:829)

vladimirpetrov avatar Jun 16 '22 12:06 vladimirpetrov

Hi @vladimirpetrov, I was able to create primary storage with Ceph. The OS of the mgmt/agents is: Operating System: Rocky Linux 8.5 (Green Obsidian) Ceph version is: ceph version 13.2.10 (564bdc4ae87418a232fc901524470e1a0f76d641) mimic (stable) CS version - 4.17.0.0

slavkap avatar Jun 16 '22 13:06 slavkap

@vladimirpetrov maybe a setup issue cc @weizhouapache @sureshanaparti ?

rohityadavcloud avatar Jul 01 '22 05:07 rohityadavcloud

I don't believe it is a setup issue because the same Ceph storage worked perfectly with CentOS 7. Please note that my Ceph version is newer than the one @slavkap tested - 15.2.16.

vladimirpetrov avatar Jul 05 '22 06:07 vladimirpetrov

Operating System: Debian 11 Ceph version is: ceph version 17.2.3 CS version - 4.17.0.1 Add primary storage failed

2022-08-13 04:28:47,358 WARN  [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:ede576d7) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a67e3adf-8acf-3ec8-9342-634bd09dc951
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-13 04:32:10,136 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Attempting to create storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 (RBD) in libvirt
2022-08-13 04:32:10,138 WARN  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 was not found running in libvirt. Need to create it.
2022-08-13 04:32:10,138 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Didn't find an existing storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 by UUID, checking for pools with duplicate paths
2022-08-13 04:32:10,206 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: rbd.cloud.priv,: Operation not supported
2022-08-13 04:32:10,206 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Failed to create the RBD storage pool, cleaning up the libvirt secret
2022-08-13 04:32:10,207 WARN  [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:0bf60b72) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a67e3adf-8acf-3ec8-9342-634bd09dc951
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-13 04:32:12,114 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Attempting to create storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 (RBD) in libvirt
2022-08-13 04:32:12,115 WARN  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 was not found running in libvirt. Need to create it.
2022-08-13 04:32:12,115 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Didn't find an existing storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 by UUID, checking for pools with duplicate paths
2022-08-13 04:32:12,183 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: rbd.cloud.priv,: Operation not supported
2022-08-13 04:32:12,184 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Failed to create the RBD storage pool, cleaning up the libvirt secret
2022-08-13 04:32:12,184 WARN  [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:0d441e25) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a67e3adf-8acf-3ec8-9342-634bd09dc951
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)

telent rbd.cloud.priv 6789 is ok telnet rbd.cloud.priv 3300 is ok create user

ceph osd pool create cloudstack
rbd pool init cloudstack
ceph auth get-or-create client.cloudstack mon 'profile rbd' osd 'profile rbd pool=cloudstack'

secret is ok but Add primary storage failed,chage the ceph version to ceph version 15.2.17 (8a82819d84cf884bd39c17e3236e0632ac146dc4) octopus (stable) on centos 7 is ok

canghai908 avatar Aug 13 '22 13:08 canghai908

I'm able to use ceph version 17.2.0 with ACS 4.16.1; @canghai908 can you check/share the logs from both mgmt server(s) and cloudstack-agent logs from the kvm host?

rohityadavcloud avatar Aug 13 '22 14:08 rohityadavcloud

@rohityadavcloud this is cloudstack-agent logs

2022-08-13 04:28:47,358 WARN  [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:ede576d7) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a67e3adf-8acf-3ec8-9342-634bd09dc951
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-13 04:32:10,136 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Attempting to create storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 (RBD) in libvirt
2022-08-13 04:32:10,138 WARN  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 was not found running in libvirt. Need to create it.
2022-08-13 04:32:10,138 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Didn't find an existing storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 by UUID, checking for pools with duplicate paths
2022-08-13 04:32:10,206 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: rbd.cloud.priv,: Operation not supported
2022-08-13 04:32:10,206 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-2:null) (logid:0bf60b72) Failed to create the RBD storage pool, cleaning up the libvirt secret
2022-08-13 04:32:10,207 WARN  [cloud.agent.Agent] (agentRequest-Handler-2:null) (logid:0bf60b72) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a67e3adf-8acf-3ec8-9342-634bd09dc951
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
2022-08-13 04:32:12,114 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Attempting to create storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 (RBD) in libvirt
2022-08-13 04:32:12,115 WARN  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 was not found running in libvirt. Need to create it.
2022-08-13 04:32:12,115 INFO  [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Didn't find an existing storage pool a67e3adf-8acf-3ec8-9342-634bd09dc951 by UUID, checking for pools with duplicate paths
2022-08-13 04:32:12,183 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Failed to create RBD storage pool: org.libvirt.LibvirtException: failed to connect to the RADOS monitor on: rbd.cloud.priv,: Operation not supported
2022-08-13 04:32:12,184 ERROR [kvm.storage.LibvirtStorageAdaptor] (agentRequest-Handler-3:null) (logid:0d441e25) Failed to create the RBD storage pool, cleaning up the libvirt secret
2022-08-13 04:32:12,184 WARN  [cloud.agent.Agent] (agentRequest-Handler-3:null) (logid:0d441e25) Caught:
com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a67e3adf-8acf-3ec8-9342-634bd09dc951
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)=

mgmt server log

2022-08-13 04:32:10,225 DEBUG [c.c.s.StorageManagerImpl] (qtp1462044018-366:ctx-3e5ccaa7 ctx-e2a6c0d8) (logid:0bf60b72) Failed to clean up storage pool: null
2022-08-13 04:32:10,225 INFO  [c.c.a.ApiServer] (qtp1462044018-366:ctx-3e5ccaa7 ctx-e2a6c0d8) (logid:0bf60b72) Failed to add data store: Failed to access storage pool
2022-08-13 04:32:10,226 DEBUG [c.c.a.ApiServlet] (qtp1462044018-366:ctx-3e5ccaa7 ctx-e2a6c0d8) (logid:0bf60b72) ===END===  172.16.3.252 -- POST  command=createStoragePool&response=json
2022-08-13 04:32:10,485 DEBUG [o.a.c.h.HAManagerImpl] (BackgroundTaskPollManager-2:ctx-fcf8bbe0) (logid:1c6fda81) HA health check task is running...
2022-08-13 04:32:12,027 DEBUG [c.c.a.ApiServlet] (qtp1462044018-18:ctx-7f80a5be) (logid:0d441e25) ===START===  172.16.3.252 -- POST  command=createStoragePool&response=json
2022-08-13 04:32:12,037 DEBUG [c.c.a.ApiServer] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) CIDRs from which account 'Acct[c5268ccf-1ac5-11ed-8fd2-525400dfc23b-admin] -- Account {"id": 2, "name": "admin", "uuid": "c5268ccf-1ac5-11ed-8fd2-525400dfc23b"}' is allowed to perform API calls: 0.0.0.0/0,::/0
2022-08-13 04:32:12,048 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) createPool Params @ scheme - rbd storageHost - rbd.cloud.priv hostPath - /cloudstack port - -1
2022-08-13 04:32:12,058 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) creating pool rv on  host 1
2022-08-13 04:32:12,062 DEBUG [c.c.a.t.Request] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) Seq 1-7861877574505333217: Sending  { Cmd , MgmtId: 90520745394747, via: 1(kvm27.cloud.priv), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.CreateStoragePoolCommand":{"_createDatastore":"false","add":"true","pool":{"id":"48","uuid":"a67e3adf-8acf-3ec8-9342-634bd09dc951","host":"rbd.cloud.priv","path":"cloudstack","userInfo":"cloudstack:AQDqdPdiP4rJEBAAttog6uNyIy9plWg0c77e_A==","port":"0","type":"RBD"},"localPath":"/mnt//0b735f43-6986-350d-b809-2d1592a7ffac","wait":"0","bypassHostMaintenance":"false"}}] }
2022-08-13 04:32:12,064 DEBUG [c.c.a.t.Request] (AgentManager-Handler-2:null) (logid:) Seq 1-7861877574505333217: Processing:  { Ans: , MgmtId: 90520745394747, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"true","details":"success","wait":"0","bypassHostMaintenance":"false"}}] }
2022-08-13 04:32:12,064 DEBUG [c.c.a.t.Request] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) Seq 1-7861877574505333217: Received:  { Ans: , MgmtId: 90520745394747, via: 1(kvm27.cloud.priv), Ver: v1, Flags: 10, { Answer } }
2022-08-13 04:32:12,064 DEBUG [c.c.a.m.AgentManagerImpl] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) Details from executing class com.cloud.agent.api.CreateStoragePoolCommand: success
2022-08-13 04:32:12,066 DEBUG [o.a.c.s.d.l.CloudStackPrimaryDataStoreLifeCycleImpl] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) In createPool Adding the pool to each of the hosts
2022-08-13 04:32:12,067 DEBUG [c.c.s.StorageManagerImpl] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) Adding pool rv to  host 1
2022-08-13 04:32:12,072 DEBUG [c.c.a.t.Request] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) Seq 1-7861877574505333218: Sending  { Cmd , MgmtId: 90520745394747, via: 1(kvm27.cloud.priv), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.ModifyStoragePoolCommand":{"add":"true","pool":{"id":"48","uuid":"a67e3adf-8acf-3ec8-9342-634bd09dc951","host":"rbd.cloud.priv","path":"cloudstack","userInfo":"cloudstack:AQDqdPdiP4rJEBAAttog6uNyIy9plWg0c77e_A==","port":"0","type":"RBD"},"localPath":"/mnt//0b735f43-6986-350d-b809-2d1592a7ffac","wait":"0","bypassHostMaintenance":"false"}}] }
2022-08-13 04:32:12,185 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) (logid:) Seq 1-7861877574505333218: Processing:  { Ans: , MgmtId: 90520745394747, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a67e3adf-8acf-3ec8-9342-634bd09dc951
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)
        at com.cloud.utils.nio.Task.call(Task.java:83)
        at com.cloud.utils.nio.Task.call(Task.java:29)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
","wait":"0","bypassHostMaintenance":"false"}}] }
2022-08-13 04:32:12,185 DEBUG [c.c.a.t.Request] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) Seq 1-7861877574505333218: Received:  { Ans: , MgmtId: 90520745394747, via: 1(kvm27.cloud.priv), Ver: v1, Flags: 10, { Answer } }
2022-08-13 04:32:12,185 DEBUG [c.c.a.m.AgentManagerImpl] (qtp1462044018-18:ctx-7f80a5be ctx-75c25dfd) (logid:0d441e25) Details from executing class com.cloud.agent.api.ModifyStoragePoolCommand: com.cloud.utils.exception.CloudRuntimeException: Failed to create storage pool: a67e3adf-8acf-3ec8-9342-634bd09dc951
        at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:672)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:350)
        at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:344)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:42)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtModifyStoragePoolCommandWrapper.execute(LibvirtModifyStoragePoolCommandWrapper.java:35)
        at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78)
        at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1830)
        at com.cloud.agent.Agent.processRequest(Agent.java:660)
        at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1080)

canghai908 avatar Aug 13 '22 15:08 canghai908

@canghai908 as the error is coming from libvirt can you please check if the ceph monitor is listening on all interfaces? Found something similar, https://listman.redhat.com/archives/libvir-list/2021-May/219715.html

shwstppr avatar Aug 16 '22 07:08 shwstppr

@shwstppr ceph monitor is listen on all interfaces.

telent rbd.cloud.priv 6789 is ok
telnet rbd.cloud.priv 3300 is ok

canghai908 avatar Aug 16 '22 12:08 canghai908

@vladimirpetrov @canghai908 your issue may be related to https://github.com/apache/cloudstack/issues/5741#issuecomment-990669423

# ceph config set mon auth_expose_insecure_global_id_reclaim false

# ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false

# ceph config set mon auth_allow_insecure_global_id_reclaim false

I was able to follow this quick install guide, apply the above workaround and was able to add ceph v17 storage to EL8/rocky linux 8: https://rohityadav.cloud/blog/ceph/

Pl re-open if you still face issue with ceph storage on EL8 @canghai908 @vladimirpetrov

rohityadavcloud avatar Aug 17 '22 08:08 rohityadavcloud

@rohityadavcloud Thanks，I wil try。

canghai908 avatar Aug 17 '22 11:08 canghai908