Add priorities to MPI queues of requests

[msimberg]

It may be useful to process events/requests according to priorities instead of submission order (or in addition to submission order).

[biddisco]

The main problem would be that once you have issued an MPI_Isend/Recv/other, then it is out of your hands at the network level. When messages are being polled, you could poll for important ones first, but whether it would make much difference, we'd have to discover by trial and error. As I mention in our weekly call, we could switch from either MPI_Test (individual), and MPI_Testany (tests potentially all the requests, but returns when the first is ready, to using MPI_Testsome - which can return a list of ready requests (ie it has to test all, so one completed couldn't sit in the queue being unhandled if other kept getting in front of it somehow). if we did this, then we could store a priority with them, and then process them in order. Alternatively, we could make sure that higher priority requests were always in the request vector at the front, so that testany would check them before others.

In general, messages will be handled in the same order they go in/out, but once async collectives are being used, lots of send/recv's might be handled before a broadcast or something and so it might help to try it. Would not be hard to implement so I'll add support once I''m done with my existing tests.

[msimberg]

Thanks @biddisco, that sounds reasonable. I realize that in the MPI case the actual latency until a request is completed would not be affected by using priorities. I suppose the main effect would be on the order that continuations would be triggered. In any case, if this is something that you want to play around with that's great, but don't worry too much about it. I just wanted to document it here before I forget it.

[biddisco]

however - if a message is high priority for processing - certainly, when the request is marked "ready" and we pass it back to the runtime - we should add it as a HP task if it is going into the task queues (and not being handled inline) ...

[msimberg]

#286 added normal and high priority queues for CUDA polling. This issue still tracks the corresponding change for MPI polling.