Polling on user-provided file descriptors

[mdorier]

Something that could be useful in Mercury for upper libraries like Margo is the possibility to inject user-provided file descriptors in the Mercury polling mechanism. Some use-cases would include injecting timerfds (to manage a set of timers that have to fire at specific time and make the progress loop return to trigger a specific callback), or injecting file descriptors through which we could make the HG_Progress call return without having received an RPC (for instance in conjunction with Argobots custom schedulers).

Some idea of API:

hg_return_t HG_Inject(hg_context_t *context, int fd, hg_cb_t callback, void* udata);

With an additional type of callback info defined as follows:

struct hg_cb_info_user_fd {
    int fd; /* file descriptor */
};

/* change to the hg_cb_info structure */
struct hg_cb_info {
    void *arg;                  /* User data */
    hg_return_t ret;            /* Return value */
    hg_cb_type_t type;          /* Callback type */
    union {                     /* Union of callback info structures */
        struct hg_cb_info_lookup lookup;
        struct hg_cb_info_forward forward;
        struct hg_cb_info_respond respond;
        struct hg_cb_info_bulk bulk;
        struct hg_cb_info_user_fd user_fd;
    } info;
};

/* change to the hg_cb_type_t enum */
typedef enum hg_cb_type {
    HG_CB_LOOKUP,       /*!< lookup callback */
    HG_CB_FORWARD,      /*!< forward callback */
    HG_CB_RESPOND,      /*!< respond callback */
    HG_CB_BULK,          /*!< bulk transfer callback */
    HG_CB_USER_FD /*!< user fd callback */
} hg_cb_type_t;

(there might be some additional relevant information, such as the type of event that was received on the file descriptor, to add to the hg_cb_info_user_fd structure)

An event on the file descriptor would trigger the user-provided callback and remove the file descriptor from the set of fds to poll from (so if the user wants to continue monitoring this fd, the callback should call HG_Inject again).

The mechanism would be sort of like the generalized MPI requests.