scverse
Add support for `sparse` arrays (pydata) to .varp and .obsp fields
Please describe your wishes and possible alternatives to achieve the desired result.
scipy.sparse only supports 2D arrays. Whereas varp and obsp support N-D arrays.
Most of the time we would want them to be sparse.
But right now anndata prevent the varp and obsp fields to contain the pydata/sparse formats.
Not all but many features are available on these matrix formats. Are there missing features that would be required as of now that would prevent this integration?
Ah, you’re talking about https://sparse.pydata.org/en/stable/.
We haven’t talked about this much yet, but it’s definitely worth discussing.
Let me know if I can help since it doesn't seem like too complicated to add and would be worth it for some applications :)
If you can point me to some of files to be looking at for updates I can have a look
@jkobject If I remember, our main stumbling block with this up until now has been the dependency on numba, which is too much for anndata.
But we could do this as an optional dep. In this case, we should think more broadly about a registration mechanism for choosing which array type gets returned from i/o. This would be helpful for GPU.
I would add that last I talked to the authors of pydata/sparse they were looking at a full rewrite. So it might be in a rocky state for a while.
@ivirshup it loooks like the package has a backend for finch now...could be worth trying out actually...would be huge for array-api compat!
Just passing by to say that @eroell and me have use cases for sparse.COO aka n-dimensional arrays also in layers. Generally, we'd be happy to see this rolled out across more slots. Finch sparse seems to be developed actively but I have not seen it get any traction yet.
@Zethson @eroell If you guys wanted to begin a PR for 3 dimensional layers/X, at least to see where the thorns are, I would not be opposed at all (and would be quite happy). You might have to handle https://github.com/scverse/anndata/issues/244 but I think all in all, this would not be that bad to deal with.