rtslib_fb: `db_root: cannot be changed because it's in use`

[discostur]

Hi,

i'm running ceph 17.2.6 via docker containers on el 8.8 and since the latest updates i get lots of kernel errors

kernel: db_root: cannot be changed because it's in use

and the target-api container is constantly rebooting:

Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: Could not load gateway: Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: Removing iSCSI target from LIO
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: No available network portal for target with iqn of 'iqn.2001-07.com.ceph:1691062051543'
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: (Gateway.load_config) successfully loaded existing target definition
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: lio disk deletion failed Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: lio disk deletion failed Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev journal[21759]: (Gateway.load_config) successfully loaded existing target definition
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev kernel: db_root: cannot be changed because it's in use
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: OSError: [Errno 22] Invalid argument
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: Traceback (most recent call last):
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 171, in _set_dbroot
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    fwrite(dbroot_path, self._default_dbroot+"\n")
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/rtslib_fb/utils.py", line 79, in fwrite
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    file_fd.write(str(string))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: OSError: [Errno 22] Invalid argument
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: Traceback (most recent call last):
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/bin/rbd-target-api", line 2930, in main
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    ceph_gw.define()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/gateway.py", line 267, in define
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    self.define_targets()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/gateway.py", line 240, in define_targets
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    target = self.define_target(iqn, target.get('ip_list', {}))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/gateway.py", line 214, in define_target
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    GWClient.define_clients(self.logger, self.config, target.iqn)
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/client.py", line 318, in define_clients
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    client.manage('present')  # ensure the client exists
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/client.py", line 609, in manage
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    client_exists = self.exists()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/client.py", line 519, in exists
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    r = lio_root.RTSRoot()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 88, in __init__
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    self._set_dbroot()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 175, in _set_dbroot
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    .format(self._default_dbroot))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: rtslib_fb.utils.RTSLibError: Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: OSError: [Errno 22] Invalid argument
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: Traceback (most recent call last):
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 171, in _set_dbroot
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    fwrite(dbroot_path, self._default_dbroot+"\n")
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/rtslib_fb/utils.py", line 79, in fwrite
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    file_fd.write(str(string))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: OSError: [Errno 22] Invalid argument
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: During handling of the above exception, another exception occurred:
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: Traceback (most recent call last):
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/bin/rbd-target-api", line 3022, in <module>
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    main()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/bin/rbd-target-api", line 2934, in main
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    ceph_gw.delete()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/gateway.py", line 351, in delete
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    lio = LIO()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/ceph_iscsi_config/lio.py", line 10, in __init__
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    self.lio_root = root.RTSRoot()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 88, in __init__
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    self._set_dbroot()
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:  File "/usr/lib/python3.6/site-packages/rtslib_fb/root.py", line 175, in _set_dbroot
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]:    .format(self._default_dbroot))
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: rtslib_fb.utils.RTSLibError: Cannot set dbroot to /var/target. Please check if this directory exists.
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: teardown: managing teardown after SIGCHLD
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: teardown: Waiting PID 80 to terminate
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: teardown: Process 80 is terminated
Aug 17 09:18:51 ceph-iscsi02-dev docker[21639]: teardown: Bye Bye, container will die with return code 0

It seems there was already a fix provided which didn't make it into the ceph container:

https://bugzilla.redhat.com/show_bug.cgi?id=1676669 https://github.com/open-iscsi/rtslib-fb/pull/156

Infos:

el 8.8 / 4.18.0-477.21.1.el8_8.x86_64 
ceph 17.2.6
docker setup

Regards, Kilian