Patrick Hoefler

Results 345 comments of Patrick Hoefler

Add partition lengths to DataFrame metadata.

This hasn't been added but it's a lot easier now with dask-expr and the new DataFrame implementation

Support for sharding when storing dask arrays to zarr

Contributions are welcome in this area!

Requested dask.distributed scheduler but no Client active

Hi, could you provide a reproducer?

Requested dask.distributed scheduler but no Client active

Can you create something with a LocalCluster instead and something that we can copy-paste?

Requested dask.distributed scheduler but no Client active

Please use a LocalCluster for a reproducer, we don't want to install docker compose to reproduce something like this

Local memory explodes on isin()

Thanks for your report. This seems to be a pandas problem and the way pandas handles the values object for Arrow Strings. FWIW, I would recommend to merge if your...

Optimizer applies parquet `filters` after loading when using `read_parquet(...).map_partitions(...).compute()`

Hi, thanks for your report. We are applying the filters when converting the dataset fragment to an arrow table. So this happens before we convert things to a DataFrame and...

Optimizer applies parquet `filters` after loading when using `read_parquet(...).map_partitions(...).compute()`

So I looked into this again and the filters are attached to the read parquet expression in both cases (sorry that pprint doesn't show it). I am closing this for...

Are there any workarounds for dask breaking altogether with higher amounts of load than what fits into a worker

Do you really need the full DataFrame back? Folks are normally writing to storage if the result is large instead of asking for everything back to the client

Are there any workarounds for dask breaking altogether with higher amounts of load than what fits into a worker

You shouldn't call compute on these large data frames, keep the data on the cluster and do your work there