flux-core icon indicating copy to clipboard operation
flux-core copied to clipboard

flux-resource: improve performance of `flux resource list`

Open grondo opened this issue 11 months ago • 1 comments

This PR improves the performance of flux resource list by modifying some really inefficient code:

  • The ResourceSetExtra wrapper class unnecessarily reinitializes the ResourceSet argument. Just stash the ResourceSet instead and forward method calls to it to avoid this wasted work.
  • resources_uniq_lines() iterates over each individual rank in each resource set to collect common lines of output together. Instead, split a resource set into all combinations of properties and iterate each of these sets. This should be the minimum number of sets required to create possibly unique lines (at least at this point). This reduces the iteration count significantly.

Timing before with a very large resource set (~10K ranks):

 time flux resource list --from-stdin < ../../large-list.json >/dev/null

real	0m6.878s
user	0m6.762s
sys	0m0.089s

Timing after these changes:

$ time src/cmd/flux resource list --from-stdin < ../../large-list.json >/dev/null

real	0m1.790s
user	0m1.662s
sys	0m0.102s

Further improvements will probably have to be made in librlist itself.

grondo avatar Mar 23 '24 17:03 grondo