Julian Samaroo

Allow the scheduler to dynamically add/remove workers

8

As discussed in #147 , it may benefit certain use cases to known when a worker is unused by Dagger entirely (specifically, no data cached on the worker) so that...

So that custom logging is easy to globally apply to a DAG

The lazy DArray has an inherent flaw: Julia's array interface was designed around eager execution, whereas the DArray wants to be lazy. This implies that the DArray doesn't compose well...

Stateless tasks are nice from a scheduling point-of-view, since we can re-schedule and rearrange tasks in any valid order. However, stateful computation allows for memory reuse, data locality, gated resource...

From https://blog.dask.org/2017/07/03/scaling

Provide some way to tell Dagger that a thunk uses multiple different kinds of processors

1

Users may use code which uses multithreading automatically, yet currently Dagger assumes that a thunk only uses one Dagger processor at a time (i.e. one CPU thread or GPU SM/CU)....

As the scheduler grows more optimizations, options, and supported features, the combination of configurations that the scheduler needs to handle correctly grows exponentially. We could do ourselves a great service...

Modern clusters (especially supercomputers) usually have more than one networking device between a given pair of nodes, and specialized network fabrics (e.g. Infiniband, NVLINK, ROCmRDMA, etc.) which can provide optimized...

Remove scheduler plugin machinery, make Sch programmable

1

The scheduler plugin system is (to my knowledge) unused by any modern users of Dagger. However, the presence of the potential for multiple external schedulers makes changing the scheduler API...

There's no need, and iterative algorithms like NMF might build up a huge-but-regular type tree, slowing down the compiler.