Deepak Cherian
Deepak Cherian
While working on my talk, I just remembered that you all had fixed a major scheduling problem that made `"cohorts"` better than `"map-reduce"` in many more cases. That isn't true...
No your fix makes my heuristics useless :P we need to update them and choose `"map-reduce"` in more cases than we do currently. The graph is significantly better and easier...
I think we should add some documentation on this stuff. We have "cloud storage buckets" under zarr( https://xarray.pydata.org/en/stable/io.html#cloud-storage-buckets) so maybe a similar section under netCDF?
> Kerchunk uses cfgrib.dataset.COORD_ATTRS in a few places to loop over possible coordinates. This doesn't contain either directionNumber, nor frequencyNumber. This could be updated. AFAICT there's no obvious way to...
> The method iris.cube.Cube.aggregated_by produces many tiny chunks and a very large graph when used with lazy data Sounds familiar to me as an Xarray user :) > and they...
> Flox looks seriously impressive and sophisticated, Thanks! > I don't think there is a prospect of maintaining something with equivalent power within Iris. But you _shouldn't_ need to. There...
> and I got pretty lost in the codec_pipeline implementation (so many iterables of tuples!) Not the first person! I _did_ made it out alive, but only barely.
A general note would be a big improvement. Is it feasible to add some `debug` level logging when doing the draw and filter?