issue when trying to start rados-target-gw

[charlieuk2k]

Hi, When trying to start rados-target-gw I get the below error, googling around has not been fruitful.

I'm running this on CentOS 7.5 with the following kernel:

Linux 4.19.2-1.el7.elrepo.x86_64

Journalctl output:

Nov 18 17:30:29 localhost.localdomain systemd[1]: Starting Setup system to export rbd images through LIO...
-- Subject: Unit rbd-target-gw.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rbd-target-gw.service has begun starting up.
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: Error in sys.excepthook:
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: Traceback (most recent call last):
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/lib/python2.7/site-packages/ceph_iscsi_config-2.6-py2.7.egg/EGG-INFO/scripts/rbd-target-gw", line 37, in exception_handler
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/lib/python2.7/site-packages/ceph_iscsi_config-2.6-py2.7.egg/EGG-INFO/scripts/rbd-target-gw", line 83, in clearconfig
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: NameError: global name 'config' is not defined
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: Original exception was:
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: Traceback (most recent call last):
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/bin/rbd-target-gw", line 5, in <module>
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: pkg_resources.run_script('ceph-iscsi-config==2.6', 'rbd-target-gw')
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 540, in run_script
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: self.require(requires)[0].run_script(script_name, ns)
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1462, in run_script
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: exec_(script_code, namespace, namespace)
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 41, in exec_
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: exec("""exec code in globs, locs""")
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "<string>", line 1, in <module>
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/lib/python2.7/site-packages/ceph_iscsi_config-2.6-py2.7.egg/EGG-INFO/scripts/rbd-target-gw", line 431, in <module>
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "build/bdist.linux-x86_64/egg/ceph_iscsi_config/common.py", line 76, in __init__
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "build/bdist.linux-x86_64/egg/ceph_iscsi_config/common.py", line 37, in __init__
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/lib/python2.7/site-packages/rados.py", line 253, in __init__
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: self.conf_read_file(conffile)
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: File "/usr/lib/python2.7/site-packages/rados.py", line 302, in conf_read_file
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: raise make_ex(ret, "error calling conf_read_file")
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: rados.Error: error calling conf_read_file: errno EINVAL
Nov 18 17:30:29 localhost.localdomain rbd-target-gw[2709]: Exception AttributeError: "'CephCluster' object has no attribute 'cluster'" in <bound method CephCluster.__del__ of <ceph_iscsi_config.common.CephCluster object at 0x
Nov 18 17:30:29 localhost.localdomain systemd[1]: rbd-target-gw.service: main process exited, code=exited, status=1/FAILURE
Nov 18 17:30:29 localhost.localdomain systemd[1]: Unit rbd-target-gw.service entered failed state.
Nov 18 17:30:29 localhost.localdomain systemd[1]: rbd-target-gw.service failed.
Nov 18 17:30:30 localhost.localdomain systemd[1]: rbd-target-gw.service holdoff time over, scheduling restart.
Nov 18 17:30:30 localhost.localdomain systemd[1]: start request repeated too quickly for rbd-target-gw.service
Nov 18 17:30:30 localhost.localdomain systemd[1]: Failed to start Setup system to export rbd images through LIO.
-- Subject: Unit rbd-target-gw.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit rbd-target-gw.service has failed.
--
-- The result is failed.
Nov 18 17:30:30 localhost.localdomain systemd[1]: Unit rbd-target-gw.service entered failed state.
Nov 18 17:30:30 localhost.localdomain systemd[1]: rbd-target-gw.service failed.

[charlieuk2k]

Also for reference, I was following these ceph docs as guidance.

[mikechristie]

Did you perform these steps:

http://docs.ceph.com/docs/mimic/rbd/iscsi-target-cli/

specifically this one:

If the Ceph iSCSI gateway is not colocated on an OSD node, then copy the Ceph configuration files, located in /etc/ceph/, from a running Ceph node in the storage cluster to the iSCSI Gateway node. The Ceph configuration files must exist on the iSCSI gateway node under /etc/ceph/.

---

On the machine you are setting up the target and running gwcli does

rbd ls

work?

Does

ceph osd lspools

have a rbd pool?

[charlieuk2k]

@mikechristie,

Yes, however the rbd-target-api fails due to rbd-target-gw not starting.

[dillaman]

@charlieuk2k Looks like you have a missing or corrupt "/etc/ceph/.conf" file.

[charlieuk2k]

Fresh install, new error! :)

Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: Error in sys.excepthook:
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: Traceback (most recent call last):
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "/usr/lib/python2.7/site-packages/ceph_iscsi_config-2.6-py2.7.egg/EGG-INFO/scripts/rbd-target-gw", line 37, in exception_handler
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "/usr/lib/python2.7/site-packages/ceph_iscsi_config-2.6-py2.7.egg/EGG-INFO/scripts/rbd-target-gw", line 83, in clearconfig
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: NameError: global name 'config' is not defined
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: Original exception was:
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: Traceback (most recent call last):
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "/usr/bin/rbd-target-gw", line 5, in <module>
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: pkg_resources.run_script('ceph-iscsi-config==2.6', 'rbd-target-gw')
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 540, in run_script
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: self.require(requires)[0].run_script(script_name, ns)
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 1462, in run_script
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: exec_(script_code, namespace, namespace)
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "/usr/lib/python2.7/site-packages/pkg_resources.py", line 41, in exec_
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: exec("""exec code in globs, locs""")
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "<string>", line 1, in <module>
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "/usr/lib/python2.7/site-packages/ceph_iscsi_config-2.6-py2.7.egg/EGG-INFO/scripts/rbd-target-gw", line 431, in <module>
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "build/bdist.linux-x86_64/egg/ceph_iscsi_config/common.py", line 82, in __init__
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: File "build/bdist.linux-x86_64/egg/ceph_iscsi_config/common.py", line 167, in init_config
Nov 20 19:34:22 iscsi13 rbd-target-gw[5639]: AttributeError: 'WriteOpCtx' object has no attribute 'new'
Nov 20 19:34:22 iscsi13 systemd[1]: rbd-target-gw.service: main process exited, code=exited, status=1/FAILURE
Nov 20 19:34:22 iscsi13 systemd[1]: Unit rbd-target-gw.service entered failed state.
Nov 20 19:34:22 iscsi13 systemd[1]: rbd-target-gw.service failed.
Nov 20 19:34:22 iscsi13 systemd[1]: rbd-target-gw.service holdoff time over, scheduling restart.
Nov 20 19:34:22 iscsi13 systemd[1]: start request repeated too quickly for rbd-target-gw.service
Nov 20 19:34:22 iscsi13 systemd[1]: Unit rbd-target-gw.service entered failed state.
Nov 20 19:34:22 iscsi13 systemd[1]: rbd-target-gw.service failed.

[mikechristie]

What version of ceph are you using on the iscsi targets? What is the value of

rpm -q python-rados

?

[charlieuk2k]

Ceph is mimic,

rpm -q python-rados=python-rados-10.2.11-0.el7.x86_64

[charlieuk2k]

Ceph -v = ceph version 10.2.11 (e4b061b47f07f583c92a050d9e84b1813a35671e)

[mikechristie]

If you are using mimic (13.x something), I think you hit an issue where the rados libs from a older el repo are being installed instead of the newer ones.

That:

rpm -q python-rados=python-rados-10.2.11-0.el7.x86_64

value is from jewel.

Do you have a older librbd too? What is the value of

rpm -q librbd1

If it is also a 10.x something then you need to update that too.

[charlieuk2k]

librbd1-10.2.11-0.el7.x86_64

Awesome, I'll see if I can get them updated.