dask-cloudprovider icon indicating copy to clipboard operation
dask-cloudprovider copied to clipboard

Add ability to specify maxSwap for ECS clusters

Open gvelchuru opened this issue 5 years ago • 1 comments

It would be nice to specify swap memory, especially when dealing with very large DataFrames

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definitions (search for maxSwap)

gvelchuru avatar Jan 28 '20 23:01 gvelchuru

I would be happy to explore this but it is worth noting that Dask manages it's own memory swapping and doesn't use the built in system swap. Futures are moved in and out of memory into the temporary worker space on disk.

If you are having trouble with workers running out of memory this may be related to your dataframe partitioning.

jacobtomlinson avatar Jan 29 '20 12:01 jacobtomlinson