flux-framework
implement PMI_PORT and support localhost:PORT and UNIX domain socket path
Problem: our simple PMI client and server only support file descriptor passing with PMI_FD but in some circumstances, file descriptor passing can be inconvenient.
MPICH implements an alternate connection mechanism in simple PMI. If PMI_PORT is set to hostname:port, then PMI tries to connect to that instead.
It might be slightly more secure to support a UNIX domain socket in addition to hostname:port. That would allow the PMI server to limit access to only one user, using SO_PEERCRED. However, that mode is not supported by MPICH so it may be useful to get feedback from MPICH developers before considering doing that, to see if they see any problem there.
Edit: this issue was forked from
- #7175
Here are a couple of references to MPICH changes where PMI_PORT was implemented. The motivating use cases over PMI_FD seem to be BLCR (kernel checkpoint/restart) and debugging with lldb and idb:
- pmodels/mpich#4339
- pmodels/mpich@5160e94fe04bb3dcee0ec17886941e8302fecccf (from SVN days before pull requests)
When PMI_PORT is used, the client performs an additional handshake
- getenv PMI_ID (hydra sets this to the rank but but the rank should be set from the value received in
initack) - send
fullinit pmiid ID - recv
initack rank=INT size=INT [appnum=INT] [spawner-jobid=STR] debug=INT
We would need to document this in the ABNF at the bottom of RFC 13. An approach to adding this to Flux would be to first implement the cilent and server to match MPICH PMI_PORT=hostname:PORT, then extend it to support PMI_PORT=/path/to/socket. This mode could be selected via a launcher option to the simple pmi plugin, like -o pmi-simple.pmi-port
FYI @raffenet or @hzhou. Let me know if I'm going down a path that is contrary to your plans for simple PMI. (Oh! I just noticed the specs in https://github.com/pmodels/mpich/tree/main/src/pmi/maint - very nice! I will have to compare that to what we've gotten by spelunking code and testing in the past)
It seems reasonable to me. I think the design should allow PMI_PORT to support a wide range of connection protocols. Because it is more of an implementation internal details when server and client are implemented together, in hydra, we just supported one protocol (tcp), which seems sufficient at the time.
Now, if we want to extend to unix sockets, I think we should extend the syntax a bit by allowing an explicit protocol. For example, tcp://host:port, unix://path, and so on. For backward compatibility we may default to tcp when it is missing.
