How do Dask-worker's get keywords?

[mrocklin]

If I ask for my workers to get a certain amount of memory or CPU usage, how do Dask workers learn about these limitations? Are we relying on the Dask workers to inspect their local VM and pull CPU counts and memory limits from there?

I ran into this problem with the dask-ssh rewrite, where I wanted to convert a large amount of Python keyword arguments, like nthreads=4 into CLI keywords, like "--nthreads 4". I'm wondering if this project faces a similar problem.

[jacobtomlinson]

I am expecting the dask workers to learn this from the container they are in.

Does that sound problematic?

[mrocklin]

Not in the common case for nthreads and memory limit, but there are some less common situations that seem to come up surprisingly commonly. People want to ...

1. Use one thread, even though they have many cores. This is common for example when using GPUs.
2. Use less common keywords like --resources or --preload

We can handle these with configuration files too, which might be easier to pass around. This probably also isn't critical starting out.

[jacobtomlinson]

Use one thread, even though they have many cores.

I guess this would come up from code that isn't threadsafe. I remember dealing with things like this in the past.

This probably also isn't critical starting out.

Ok I'll leave this issue here but not worry about it immediately.

[gvelchuru]

I guess this would come up from code that isn't threadsafe. I remember dealing with things like this in the past.

Not only code that isn't threadsafe, I could also use this since my fargate workers don't have enough memory to run 4 threads concurrently.

[jacobtomlinson]

@gvelchuru have you selected the maximum CPU and memory on Fargate and found that you do not have enough memory?

[gvelchuru]

@gvelchuru have you selected the maximum CPU and memory on Fargate and found that you do not have enough memory?

Yes

[jacobtomlinson]

Does reducing the number of vCPU help?

[gvelchuru]

When I reduce the number of vCPUs I also need to reduce the amount of memory, which puts me in the same situation. For example, if I want a single worker with 30 GB memory, I cannot do that because the maximum memory for 1 vCPU is 8GB.

[jacobtomlinson]

Sorry for the delay here, I was out for a while and have only just managed to get back here.

Are you still having trouble with this @gvelchuru ?

[gvelchuru]

I ended up changing my workflow to get around this, but it could be useful in general for Fargate support.