flux-framework
resource: supporting "external" resources outside the direct ownership of Flux
It would be great for resources that can't be "owned" by Flux to support querying the availability of those resources and their acquisition. An example where this could be useful is licenses that are shared between clusters and desktop users (e.g., Matlab). Another example could be black-box vendor systems that Flux has limited or no visibility into, like AWS's spot instances.
The downside being that there would be a significant performance overhead when attempting to schedule jobs that request them. They would have to be used judiciously.
This reminds me of a resource monitoring work that we need to do with flux-core. All of the compute nodes will be marked as "no available" to begin with and as a monitoring system sends a resource event on a node (or nodes) to the scheduler, it can make use it (or them) for scheduling.
This would be a "push model" in that an extern component push resource events to the scheduler. Do you think a push model works for your case as well? It seems like "pull" model in which the scheduler will go out and query the availability would be less precise and lead to resource state inconsistency?
Do you think a push model works for your case as well?
Maybe, but it'll be racy. For example, if the license server reports that X licenses are available, then a little bit later gives out Y of them and sends a message to Flux, while that message is in flight, Flux could allocate a job that requires all X licenses (which will ultimately fail when the job goes to run and acquire the licenses).
I guess the "pull" model can be racy as well. While attempting to schedule a job requiring X licenses, Flux could ping the license server if it has X free licenses and get a positive response then resume finding the remaining resources for the job, when Flux finally goes to allocate the selected set of resources, the license server could have handed out licenses in the meantime. That first message will instead have to be a "see if X licenses are free, and if so allocate/reserve them immediately" which would then have to be undone if the job isn't currently satisfiable due to other resources.
One option: we could add support for returning a job to the queue after an allocation is made if Flux/the job fails to acquire these "external" resources due to races/resource state inconsistencies.
Certainly not the cleanest solution though. Hmmm.
To follow up on a face-to-face @dongahn and I had at ECP annual:
One option: we could add support for returning a job to the queue after an allocation is made if Flux/the job fails to acquire these "external" resources due to races/resource state inconsistencies.
We will need support for this regardless of licenses etc. If a compute node goes down during the time between allocation and the job starting, we will want to have this behavior to fall back on. Could also be applicable for when the prologue runs and determines the node is in a "bad state" and thus unusable (but not totally down).