Tom Nicholas

Defer to merge_chunks in special cases of rechunk

4

#221 introduced `merge_chunks`, a special-case of `rechunk` that can be implemented using `blockwise`. I noticed that whilst `reduction` calls `merge_chunks` directly, inside `ops.rechunk` the [primitive rechunk is always called](https://github.com/tomwhite/cubed/blob/93ad984e7b0445164ab11b3c3f3a3b7db6c3bc97/cubed/core/ops.py#L631C11-L631C11). Shouldn't...

We briefly discussed the difficulty of maintaining multiple executors. In https://github.com/tomwhite/cubed/pull/168#issuecomment-1542892933 I suggested expanding the CI to run different test jobs with different executors installed. I also saw [this dask...

Pangeo Climate Anomalies example

9

I tried to set up the problem of calculating the anomaly with respect to the group mean from https://github.com/pangeo-data/distributed-array-examples/issues/4. I used fake data with the same chunking scheme instead of...

Creating cubed arrays from lazy xarray data

8

To subset my data whilst getting around #196 I tried slicing using xarray's lazy indexing machinery before converting to cubed arrays using `.chunk` (a trick which when used with dask...

Naming tasks by passing xarray metadata

2

I was looking deeper into how to make https://github.com/pydata/xarray/issues/7813 work. So looks like the nodes are named when they are created by `Plan._new`. Q's: - Would it make sense to...

AsyncModalDagExecutor raises when run from a jupyter notebook

1

 returns ```python ╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮ │ /tmp/ipykernel_495009/1422487590.py:1 in │ │ │ │ │ │ /home/tom/Documents/Work/Code/cubed/cubed/core/array.py:112 in compute │ │ │ │ 111...

Hypothesis testing of different chunking patterns

1

FYI this might be useful in cubed's tests too https://github.com/dask/dask/pull/9374

Rename package to something more unique?

2

@yuvipanda pointed out that the name "cubed" is almost impossible to Google - there are so many other projects called cubed. (In our field "cubed" would also likely get confused...

- [x] Closes #9016 - [x] Tests added - [ ] User visible changes (including notable bug fixes) are documented in `whats-new.rst` - [ ] New functions/methods are listed in...

Using hidefix to determine byte ranges in HDF files?

2

I'm building [VirtualiZarr](https://github.com/TomNicholas/VirtualiZarr), an evolution of [kerchunk](https://github.com/fsspec/kerchunk), that allows you to determine byte ranges of chunks in netCDF files, but then concatenate the virtual representation of those chunks using xarray's...