conf_url_rados: verifying reload of ganesha's exports triggered by Ceph's object watch-notify mechanism

[ajarr]

By setting watch_url in RADOS_URLS config block, we can have nfs-ganesha watch a Ceph RADOS object. When an application (e.g., ceph's mgr/nfs module) wants ganesha to dynamically reload its exports, it notifies the watched RADOS object and makes nfs-ganesha send itself a SIGHUP. The watch callback handler, 'rados_url_watchcb', sends ack back to the notifier before sending SIGHUP to reload ganesha's exports [1]. This means that the application isn't aware of the result of the config reload. This is unlike DBus Add/Remove/Update Export interfaces that sends success or error message back to the application.

Jeff Layton suggested that the watch callback handler could directly call reread_config() and send back errors. This will mean that the watch callback will block and wait for some event. The change won't be trivial as possible deadlocks will need to be avoided. The Ceph context where the watch callback runs, and the librados mutexes held in it will need to be considered.

Alternatively, Jeff suggested that the application could use DBus ShowExports query to get the current state of ganesha's exports. However, the application (mgr/nfs module) may run in an environment that does not want to use DBus service, e.g., OpenStack [2].

[1] https://github.com/nfs-ganesha/nfs-ganesha/commit/b75f5f84fc#diff-bd5efdcc946acd67af5f6149d67c6d651a6f4abb6cbb8823fba167e39fca6c32R364 [2] https://blueprints.launchpad.net/manila/+spec/cephfs-nfs-drop-dbus

[ffilz]

What would the mechanism for sending back errors be? Could blocking issues on the librados side be solved by dispatching a separate thread to perform the reread_config() and send the errors back?

The reread_config() process now uses a mutex to serialize so that would need to be considered.

What opportunities for deadlock are possible?

[ajarr]

What would the mechanism for sending back errors be? Could blocking issues on the librados side be solved by dispatching a separate thread to perform the reread_config() and send the errors back?

From #cephfs IRC channel on OFTC...

jlayton: See rados_notify2() and rados_notify_ack() plus all of the stuff that decodes the responses (which you'll now want to do). so I guess the program that sends these notifications will need to call those and decode the responses.

The reread_config() process now uses a mutex to serialize so that would need to be considered.

What opportunities for deadlock are possible?