Tom Nicholas

There is an interesting [issue on safetensors](https://github.com/huggingface/safetensors/issues/532) about "multi-part uploads". Apparently it's not officially supported but nevertheless widespread. This suggests desire for the model weights to be chunked and/or version-controlled,...

Creating persisted virtual references looks like: (data engineer code) ```python # reading into in-memory ChunkManifest objects vds = open_virtual_dataset(, reader='safetensors') # serializing references to some kind of persistent storage format...

@nenb a rough plan: 1) Write an virtualizarr reader for safetensors format - Resist temptation to support wide range of features 2) Dtype support? 3) Download some example safetensors and...

Similarity to netCDF3: - IIUC safetensors are not chunked or compressed - which means that it's analogous to netCDF3 - So you have to choose an arbitrary chunking scheme up...

Great idea! There could also be different tabs for examples using different executors.

Another idea from that discussion was to animate the Plan itself. - The arrays themselves could be squares like in the image above, or potentially even cubes like in the...

Here's the version from today 

Yes to all of these! Also an example showing that xarray groupby with flox works.

> ``` > # this should pass, since c can be fused into an op that takes 700 bytes > # (ones is 100, conversion to int32 is 400, conversion...

I had forgotten about this, and have been explaining it to people as if this was how it already works: > move the memory checks to when the FinalizedPlan is...