Lawrence Mitchell

`from_array` in your example has to build the full 40GiB array on the client side, that is, as a single allocation, and then move it to the workers. Since your...

> "Best practice is to generate the chunked array directly on the workers" can you give an example of how to do that? Use one of the [creation mechanisms](https://docs.dask.org/en/stable/array-creation.html#) that...

OK, in this case, I recommend using [`segysak`](https://github.com/trhallam/segysak) to load the SEG-Y files and then interface with dask. They have an [example doing just this](https://segysak.readthedocs.io/en/latest/examples/example_segysak_dask.html) in their documentation, which should...

I am still a little confused, `da.from_array` delivers an array on the workers, no? Why are you then doing `client.scatter` with it?

> That didn't work, but setting the environment variable `export DASK_DISTRIBUTED__DIAGNOSTICS__NVML=False` did. Thanks for pointing me in the right direction. This usually indicates a bug in the way dask config...

> I'll try and find some time to handle this properly in distributed. dask/distributed#6678

Short-term fix disabling compression is in #957.

So, suppose that there are 4 GPUs in the system, but `CUDA_VISIBLE_DEVICES=1,3`. Then: ```python from cuda import cuda cuda.cuInit(0) cuda.cuDeviceCanAccessPeer(0, 1) # can device 1 talk to device 3? =>...

> The easiest way to test is actually to just test with UCX-Py between peers. IIUC: ``` # let both processes see both devices $ CUDA_VISIBLE_DEVICES=0,1 python send-recv-core.py --reuse-alloc -d...

If I do: ``` $ dask-scheduler --protocol ucx --scheduler-file foo.json & $ for i in $(seq 0 7); do CUDA_VISIBLE_DEVICES=$i dask-cuda-worker --scheduler-file foo.json --protocol ucx &; done $ python local_cudf_merge.py...