Trouble: gwcli.utils.GatewayAPIError: Unknown error connecting to http://localhost:5001/api/config

[Phoenix500526]

Please help me as this is stopping me cold.

In accordance with the document [CONFIGURING THE ISCSI TARGET USING THE COMMAND LINE INTERFACE](Configuring the iSCSI Target using the Command Line Interface — Ceph Documentation) , I have installed ceph-iscsi, python-rtslib, targetcli and tcmu-runner. When I tried to use gwcli to configure the iSCSI target, I got some error messages:

$[root@node1:~] gwcli -d
Adding ceph cluster 'ceph' to the UI
Fetching ceph osd information
Querying ceph for state information
Traceback (most recent call last):
  File "/usr/bin/gwcli", line 194, in <module>
    main()
  File "/usr/bin/gwcli", line 105, in main
    root_node.refresh()
  File "/usr/lib/python2.7/site-packages/ceph_iscsi-3.4-py2.7.egg/gwcli/gateway.py", line 62, in refresh
  File "/usr/lib/python2.7/site-packages/ceph_iscsi-3.4-py2.7.egg/gwcli/gateway.py", line 99, in _get_config
  File "utils.py", line 451, in __getattr__
gwcli.utils.GatewayAPIError: Unknown error connecting to http://localhost:5001/api/config

In the first instance, I have checked the status of those three services: rbd-target-api, rbd-target-gw and tcmu-runner. All of them are running well. Then, I tried to use curl command to get config information fromhttp://localhost:5001/api/config, and it worked well too. The config information is as follows:

[root@node1:~]$ curl --insecure --user admin:admin -X GET http://localhost:5001/api/config
{
  "created": "2021/07/12 07:45:31", 
  "discovery_auth": {
    "mutual_password": "", 
    "mutual_password_encryption_enabled": false, 
    "mutual_username": "", 
    "password": "", 
    "password_encryption_enabled": false, 
    "username": ""
  }, 
  "disks": {}, 
  "epoch": 0, 
  "gateways": {}, 
  "targets": {}, 
  "updated": "", 
  "version": 11
}

And here is something strange. From the console message, the execution of gwcli command failed, but from the log message that printed in /var/log/rbd-target-api/rbd-target-api.log, the command seems perform well. The rbd-target-api.log is shown down below.

# /var/log/rbd-target-api/rbd-target-api.log

# before running gwcli command
2021-07-13 14:06:48,933     INFO [_internal.py:87:_log()] - ::ffff:127.0.0.1 - - [13/Jul/2021 14:06:48] "POST / HTTP/1.1" 404 -
2021-07-13 14:06:48,935     INFO [_internal.py:87:_log()] - ::ffff:127.0.0.1 - - [13/Jul/2021 14:06:48] "POST / HTTP/1.1" 404 -

# after running gwcli command
2021-07-13 14:06:51,359    DEBUG [common.py:505:refresh()] - config refresh - current config is {u'updated': u'', u'created': u'2021/07/12 07:45:31', u'disks': {}, u'epoch': 0, u'version': 11, u'gateways': {}, u'targets': {}, u'discovery_auth': {u'username': u'', u'password_encryption_enabled': False, u'mutual_username': u'', u'mutual_password_encryption_enabled': False, u'mutual_password': u'', u'password': u''}}
2021-07-13 14:06:51,359    DEBUG [common.py:139:_open_ioctx()] - (_open_ioctx) Opening connection to rbd pool
2021-07-13 14:06:51,360    DEBUG [common.py:146:_open_ioctx()] - (_open_ioctx) connection opened
2021-07-13 14:06:51,360    DEBUG [common.py:118:_read_config_object()] - _read_config_object reading the config object
2021-07-13 14:06:51,361    DEBUG [common.py:168:_get_ceph_config()] - (_get_rbd_config) config object contains '{
    "created": "2021/07/12 07:45:31",
    "discovery_auth": {
        "mutual_password": "",
        "mutual_password_encryption_enabled": false,
        "mutual_username": "",
        "password": "",
        "password_encryption_enabled": false,
        "username": ""
    },
    "disks": {},
    "epoch": 0,
    "gateways": {},
    "targets": {},
    "updated": "",
    "version": 11
}'
2021-07-13 14:06:51,362     INFO [_internal.py:87:_log()] - ::1 - - [13/Jul/2021 14:06:51] "GET /api/config HTTP/1.1" 200 -
2021-07-13 14:06:54,256     INFO [_internal.py:87:_log()] - ::ffff:192.168.15.62 - - [13/Jul/2021 14:06:54] "GET /api/config HTTP/1.1" 403 -
2021-07-13 14:06:54,258     INFO [_internal.py:87:_log()] - ::ffff:192.168.15.62 - - [13/Jul/2021 14:06:54] "GET /api/config HTTP/1.1" 403 -
2021-07-13 14:06:54,261     INFO [_internal.py:87:_log()] - ::ffff:192.168.15.62 - - [13/Jul/2021 14:06:54] "GET /api/config HTTP/1.1" 403 -
2021-07-13 14:06:54,263     INFO [_internal.py:87:_log()] - ::ffff:192.168.15.62 - - [13/Jul/2021 14:06:54] "GET /api/config HTTP/1.1" 403 -
2021-07-13 14:06:54,265     INFO [_internal.py:87:_log()] - ::ffff:192.168.15.62 - - [13/Jul/2021 14:06:54] "GET /api/config HTTP/1.1" 403 -

As you can see above, the rbd-target-api daemon can receive the get configuration request from gwcli. The http status code is 200 and the daemon also generated the config object. It seems that something went wrong in gwcli ?

In addition，I also tried to restart the rbd-target-api service but it changed nothing.

My environment is as follows:

• OS：Centos 7 Linux 5.11.16-1.el7.elrepo.x86_64 x86_64
• application version：
  • ceph-iscsi: 3.4-1.el7
  • gwcli: 2.7
  • tcmu-runner: 1.4.0-rc1
  • python-rtslib: 2.1.74-1.el7_9

My iscsi-gateway.cfg is shown below and I'm ensure that this file has been copied to node1, node2 and node3. node1、node2 and node3 are three remote servers.

$ cat /etc/ceph/iscsi-gateway.cfg 
[config]
# Name of the Ceph storage cluster. A suitable Ceph configuration file allowing
# access to the Ceph storage cluster from the gateway node is required, if not
# colocated on an OSD node.
cluster_name = ceph

# Place a copy of the ceph cluster's admin keyring in the gateway's /etc/ceph
# drectory and reference the filename here
gateway_keyring = ceph.client.admin.keyring

# API settings.
# The API supports a number of options that allow you to tailor it to your
# local environment. If you want to run the API under https, you will need to
# create cert/key files that are compatible for each iSCSI gateway node, that is
# not locked to a specific node. SSL cert and key files *must* be called
# 'iscsi-gateway.crt' and 'iscsi-gateway.key' and placed in the '/etc/ceph/' directory
# on *each* gateway node. With the SSL files in place, you can use 'api_secure = true'
# to switch to https mode.

# To support the API, the bear minimum settings are:
api_secure = false

# Additional API configuration options are as follows, defaults shown.
api_user = admin
api_password = admin
api_port = 5001
trusted_ip_list = node1,node2,node3

This question has been bothering me for days. Can anyone give me some hints or suggestions?

Thanks in advance.