flux-core
flux-core copied to clipboard
flux-resource: improve performance of `flux resource list`
This PR improves the performance of flux resource list
by modifying some really inefficient code:
- The
ResourceSetExtra
wrapper class unnecessarily reinitializes theResourceSet
argument. Just stash theResourceSet
instead and forward method calls to it to avoid this wasted work. -
resources_uniq_lines()
iterates over each individual rank in each resource set to collect common lines of output together. Instead, split a resource set into all combinations of properties and iterate each of these sets. This should be the minimum number of sets required to create possibly unique lines (at least at this point). This reduces the iteration count significantly.
Timing before with a very large resource set (~10K ranks):
time flux resource list --from-stdin < ../../large-list.json >/dev/null
real 0m6.878s
user 0m6.762s
sys 0m0.089s
Timing after these changes:
$ time src/cmd/flux resource list --from-stdin < ../../large-list.json >/dev/null
real 0m1.790s
user 0m1.662s
sys 0m0.102s
Further improvements will probably have to be made in librlist itself.