Document best practices for integrating eventloops

[vchuravy]

@dongahn and I chatted about this during SC. In https://github.com/flux-framework/FluxRM.jl/pull/17 I started integrating Flux with Julia libuv based event-loop. Right now I am still running into hangs that I haven't been able to narrow down.

I think my wishlist for documentation would be:

• What must a external event loop do to guarantee progress
• Calling into Flux from a multi-threaded program
  • On which thread are callbacks executed
• IIRC there are two different modes are reactor can run in, and I am still a bit confused about that.
• Clear definition of blocking and non-blocking operations and how long external data needs to be kept alive for non-blocking

My goal in the end is to never be blocked in Flux and have a rather tight integration with the Julia event loop. One complication is that Julia supports task-migration and so the question is if there is any thread-local data on the Flux I might run against.

[garlick]

A couple of quick answers:

If the flux reactor is the inner reactor loop, the only straighforward way to embed IMHO it is to register a periodic timer in the outer event loop that calls flux_reactor_run (FLUX_REACTOR_NOWAIT), effectively busy-polling the flux loop. This guarantees that everybody makes progress in a purely reactive setting, but unfortunately one is forced to choose the polling period to balance wasted CPU versus responsiveness to Flux events and timeout inaccuracies (Flux does not embed any timeouts in its API AFAIK so I don't think there are any built in constraints on the polling period). Unfortunately this is not ideal and not what I would call a tight integration. The alternative would be for Flux to expose its internal event sources and timeout values (or fold them up into an eventfd(2)) so that the outer event loop could trigger on those sources as well as its own. That's unfortunately quite a bit of work as we currently embed libev and completely hand the event sources off for it to manage.

On multiple threads, a key constraint is that the flux_t handle cannot be used by multiple threads concurrently. The lowest level functions for moving messages between queues and the file descriptor are not re-entrant so this would lead to message corruption and segfaults. Note that a flux_future_t created by flux_rpc() and variants takes a reference on the flux_t handle and stores it in the future, so futures must be handled in the same thread that created them.

The flux_t and flux_future_t objects implicitly use the reactor, which perhaps makes this more opaque than it should be. When you register a message handler on a flux_t handle, you are handing it off to message dispatcher that runs within a reactor watcher for the broker socket. When you call flux_rpc() and get back a future, you are registering a message handler for the response, which does the same. Message handler callbacks and future continuation callbacks are made by the reactor, directly or indirectly. Therefore the reactor has to run in order for callbacks to be made, and the callbacks are always made in the thread that is running the reactor.

The safest way to do concurrent flux things in a multi-thread program is for each thread to have its own flux_t handle and flux reactor. This is how the flux broker works, so this idiom is well tested. In the flux broker, all synchronization between threads is with messages, except during thread creation and destruction. I realize that's likely not particularly helpful advice to someone developing a language binding.

Flux does make extensive use of POSIX errno which I think is implemented as thread-specific data. Not sure if that presents a problem for Julia's task migration.