Carter Francis

I started a PR for adding in support for `zarr` here #2798. I think this is a nice feature to offer, especially if someone is operating with larger datasets and...

@din14970 > I looked briefly at #2617 but there is a lot there, so I won't pretend I fully understand everything you tried to do. I would guess that solving...

> Is anyone else working on this? @CSSFrancis? If not, I have some (unexpected) development time this weekend, and would really like to get this functionality completed. So that I...

From my understanding there are really two parts to this problem which are increasingly intertwined. This is how to handle I-O and running a `map` like function or UDF. These...

> Ah I forgot, and using multiprocessing by default. That removes the GIL from the equation and similar restrictions in the HDF5 library as an example. Even though many NumPy...

> * I agree about `zarr`. It is so much faster! I'll probably start using it for all of my data processing. > - However, the "folder structure" will be...

> * There is no such thing as a chunking strategy that is reasonable for most needs. HyperSpy currently uses chunks that contain at least one full signal, but this...

Just a note that I think you need to have answered a certain amount of questions to make a tag. Not a huge problem but if no one here can...

Maybe this is too abstract of a class idea, but I could see the `Decomposition` class as a general boilerplate for any simplified representation for some data. This is very...

It seems like this might be an issue with numpy trying to force an array to be nonragged. Id imagine the problem is when we cast everything to a dask...