cubed
cubed copied to clipboard
Change Coiled executor to use Dask async API
It would be nice if we could change CoiledFunctionsDagExecutor to be similar to (or based on) AsyncDaskDistributedExecutor, as it would then inherit all the features of that latter, like retries, backups, batching etc (see #276).
When doing #309 I noticed that the underlying Dask client used for coiled.function is not asynchronous. So to do this PR we'd need to get the cluster object from the function (see https://docs.coiled.io/_modules/coiled/function.html), then construct an asynchronous client ourselves. The code would then look very like AsyncDaskDistributedExecutor.