anndata icon indicating copy to clipboard operation
anndata copied to clipboard

[WIP] Support for scipy.sparse.{csr,csc}_array

Open ivirshup opened this issue 2 years ago • 1 comments

Starting support for scipy.sparse csc_array and csr_array. These are the new array-like interfaces for sparse matrices from scipy.

Fixes #746, #859

Still very WIP

TODO:

  • [ ] Figure out compat with scipy<1.11
  • [ ] Demo of how to load an array from disk
  • [ ] Support for .X
  • [ ] Tests (figure out which ones need adding)
  • [ ] Fix concatenation tests
  • [ ] Fix view tests
  • [ ] Test for indices dtype after concatenation (e.g. #859)
  • [ ] Release note

ivirshup avatar Jun 27 '23 12:06 ivirshup

Codecov Report

Attention: Patch coverage is 80.00000% with 25 lines in your changes are missing coverage. Please review.

Project coverage is 83.89%. Comparing base (0a768fc) to head (10ad4c9).

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1028      +/-   ##
==========================================
- Coverage   84.18%   83.89%   -0.29%     
==========================================
  Files          35       35              
  Lines        5607     5675      +68     
==========================================
+ Hits         4720     4761      +41     
- Misses        887      914      +27     
Files Coverage Δ
src/anndata/_core/index.py 93.24% <100.00%> (+0.04%) :arrow_up:
src/anndata/_core/views.py 84.93% <100.00%> (+0.87%) :arrow_up:
src/anndata/_io/specs/methods.py 88.00% <ø> (ø)
src/anndata/utils.py 84.12% <100.00%> (ø)
src/anndata/_core/anndata.py 84.71% <81.81%> (-0.12%) :arrow_down:
src/anndata/_core/sparse_dataset.py 94.09% <85.71%> (-0.63%) :arrow_down:
src/anndata/compat/__init__.py 81.73% <75.00%> (-1.60%) :arrow_down:
src/anndata/_core/merge.py 82.72% <82.60%> (-0.20%) :arrow_down:
src/anndata/tests/helpers.py 83.81% <63.15%> (-2.36%) :arrow_down:

... and 2 files with indirect coverage changes

codecov[bot] avatar Jun 27 '23 12:06 codecov[bot]

  • [x] backed sparse array (instead of matrix)

ilan-gold avatar Mar 28 '24 12:03 ilan-gold

Ok @ivirshup I think this is read for a review. This PR should only focus on the use of the object in the AnnData class and not on reading. That will come next, but here there is writing functionality (which made sense to me to put here)..

ilan-gold avatar Apr 05 '24 14:04 ilan-gold

The test failures are from "FAILED anndata/tests/test_backed_sparse.py::test_append_overflow_check[csr_array-zarr] - numpy.core._exceptions._ArrayMemoryError: Unable to allocate 16.0 GiB for a..." - we could xfail these as well, but it seems like since this works for matrix, maybe there's a bug here?

ilan-gold avatar Apr 08 '24 10:04 ilan-gold

sorry for the force push, VS code messed things up and I simply undid the mess to get a clean merge commit.

flying-sheep avatar Apr 09 '24 11:04 flying-sheep

No worries @flying-sheep - much appreciated!

ilan-gold avatar Apr 09 '24 12:04 ilan-gold

Demo of how to load an array from disk

Probably belongs in the follow up PR given that we did not implement that functionality here

ilan-gold avatar Apr 10 '24 08:04 ilan-gold