pyDVL
pyDVL copied to clipboard
More fine-grained assignment of resources to tasks in MapReduceJob
In order to have proper interaction with within-node parallelization of utility computations and avoid cpu contention, we need to be able to configure remote tasks to run e.g. one per node, or to limit the number of cores (and ram and gpus) that they see.
We might want to drop MapReduce altogether (think of issues with convergence criteria)
Usage of MapReduce is deprecated, fine grain config is achieved in #558 via joblib.