Jim Crist-Harif

From looking at the code, it looks like passing `memory_limit` to `LocalCluster` sets the memory limit per worker (it's forwarded directly), not total cluster. This also matches the observed behavior:...

I just noticed that too. This is because we take `min` of the user input and the total available system memory. https://github.com/dask/distributed/blob/defe454f63199799b403a3ddeee04b473adf0dfd/distributed/worker.py#L3805 So if your machine has 16 GiB of...

The code can be found here: https://github.com/dask/distributed/blob/main/distributed/system.py. It should take into account cgroups limits and RSS rlimit settings, but maybe slurm doesn't set those? Dask's memory accounting also isn't perfect,...

> Should I be using dask-jobqueue instead of dask.distributed, then? I've not done so because it looks like jobqueue is designed for a higher level of control; it controls the...

Setting `verify_meta=False` will silence these errors, but will lead to issues downstream since some of the partitions don't match the metadata. I asked Paul to raise this issue, since I...

Thanks for opening this! This seems like a useful case to handle. *Aside: I'm curious to learn more about your use case here (if you're willing to share). Is this...

> So would need to do some custom hacky parsing it feels like. Any suggestion for this? Yeah, this seems unpleasant. The good news is the feature I outlined above...

> I made a workaround for now to be able to test the scenario Glad to hear it! I'm curions - how'd the test turn out? Do you still think...

I'm sorry, I'm not sure I understand this issue? What are you trying to do here? > but if you want to create a Decoder that will handle a Union...

Sorry, there's _a lot_ above, I'll try to respond to what I think are your current issues. > Like, does a struct always have to be the outer most decoding...