cubed icon indicating copy to clipboard operation
cubed copied to clipboard

Change Coiled executor to use Dask async API

Open tomwhite opened this issue 2 years ago • 1 comments

It would be nice if we could change CoiledFunctionsDagExecutor to be similar to (or based on) AsyncDaskDistributedExecutor, as it would then inherit all the features of that latter, like retries, backups, batching etc (see #276).

tomwhite avatar Sep 05 '23 12:09 tomwhite

When doing #309 I noticed that the underlying Dask client used for coiled.function is not asynchronous. So to do this PR we'd need to get the cluster object from the function (see https://docs.coiled.io/_modules/coiled/function.html), then construct an asynchronous client ourselves. The code would then look very like AsyncDaskDistributedExecutor.

tomwhite avatar Sep 13 '23 15:09 tomwhite