HG: Context/class destruction and finalization of servers

[frostedcmos]

We are encountering a problem in DAOS with how to properly finalize/shutdown servers using APIs that mercury provides today.

The root of the problem is that in order to cleanly shutdown DAOS we need to destroy all contexts/hg classes/endpoints. As we start to destroy endpoint-associated structures and issuing HG_Context_destroy() an RPC from a client or other servers could arrive at that time. If that happens, HG_Context_destroy() would return an error indicating that not all handles are released yet.

One approach to solve this would be to introduce new API such as HG_Context_disable() which would shut down the underlying endpoint (so no new rpcs can come), without destroying any associated structures.

With that the shutdown sequence could be done as:

• Disable endpoint
• Flush/process all remaining RPCs that arrived to that endpoint
• Destroy the context/associated endpoint/structures.

The other approach we've tried is to disable all progressing of contexts before we perform flush/context destroy, however that only seems to work for providers that do not perform automatic progress on their own.