unidist
unidist copied to clipboard
modin-project
Unified Distributed Execution
I was unable to reproduce exactly the same problem without Modin. KeyError reproduced when called `query_compiler.to_pandas` from worker process (during pickling) with changes from https://github.com/modin-project/modin/pull/6673 ```python import modin.pandas as pd...
We should figure out why MPI is slow here and fix it. https://github.com/intel-ai/timedf_benchmarks/blob/b092ea0d490eb630224fc4ffdbc2f62630f57e49/timedf_benchmarks/hm_fashion_recs/fe.py#L159
We should figure out why the slow down occurs and fix it. https://github.com/intel-ai/timedf_benchmarks/blob/b092ea0d490eb630224fc4ffdbc2f62630f57e49/timedf_benchmarks/hm_fashion_recs/fe.py#L228
MSMPI implements a subset of features from MPI 3.1 standard, one of which is MPI shared memory - https://github.com/Microsoft/Microsoft-MPI#version-of-mpi-standard. We should enable the MPI shared object store for msmpi.
When putting data into the MPI shared object store we should try to use aligned memory access. This should improve performance.
Actor tests for Ray hang in CI for some reason but pass locally.
Unidist returns a result, but one of the workers cannot finish the job. ```python import unidist import time @unidist.remote def g(number): time.sleep(1) return number**2 @unidist.remote def f(): results = []...
Parameters support should be added to `unidist.init` method to give more flexibility in setup framework. Parameters are the next: - num_cpus - backend - address - ...
Spawn a background thread which send data to workers, So the main thread is not blocked during the data send and serialisation processes.
Alternative to the current round robin approach, A possible way of scheduling is to track the number of tasks running on each worker in a scheduler class. This could be...