Patrick Hoefler

Results 345 comments of Patrick Hoefler

Are there any workarounds for dask breaking altogether with higher amounts of load than what fits into a worker

I am closing this for now, calling compute will always load everything into memory

bug: handle list like objects i.e. `pd.Index` in `_simplify_down`

Can you provide a reproducer when this happens? Generally, we want to sanitize the input earlier so that we only get down there with proper lists

bug: handle list like objects i.e. `pd.Index` in `_simplify_down`

Can you post a snippet of the xarray code you are running that reproduces this?

Use Task class instead of tuple

> That would force materialization client side as well... which may be ok... I have to think about this a little more. Yeah I think maybe we only do this...

Ensure persisted collections are released without GC

thx

map_overlap passes wrong block_info[:]['array-location']

Thx! Yes this should change, the fix is not trivial unfortunately

Improve how normalize_chunks selects chunk sizes if auto is given

Yes, that is my current idea, but I haven't looked into the previous_chunks=None case very closely yet. I don't see a reason why this should be affected though. I mostly...

KilledWorker (exceeded 95% memory budget) with new optimizer

Hi, thanks for your report. Quick question about your cluster spec, your workers seem small, I would recommend providing a bit more memory, i.e. 4 GB per thread is what...

KilledWorker (exceeded 95% memory budget) with new optimizer

> with memory_limit='30GB' it has 15Gb per thread and still runs out of memory. Sorry that's on me, I should have read the docs. I assumed it was for the...

Improve how normalize_chunks selects chunk sizes if auto is given

I can patch this temporary if necessary. Let me know if you can patch this on your end without issues for users