Ilan Gold

@dicklim Can you make a smaller and reproducible example? Also could you add more details of your environment? For example: ```python import scanpy as sc adata = sc.datasets.pbmc3k() sc.pp.scrublet(adata, n_neighbors=10)...

> Is there a sparse equivalent to concat these arrays? What do you mean by this? Equivalent to what?

I don't think the arrays are zero-dimensional - that's just the errror message: ```python import scipy as sp import numpy as np X = sp.sparse.random(10, 10, format="csr") np.concatenate([X, X]) ---------------------------------------------------------------------------...

@Illviljan Thanks, but unfortunately, that's not an option for us.

@phofl Is there a plan for this? If you could give a quick outline of what needs to change, I could take it from there?

@phofl Perhaps you could point me to the refactoring PR that you mentioned earlier? You seem to have some sense that this PR may have caused this issue.

@c0nleyinnnn This issue is still open with no PR, so we have not started looking into it here, although I've begun work in `anndata`. As I mentioned previously, MLX (Apple's...

@pentschev Apologies for the bad math, shameful on my part. However, there are still some things I don't understand that are affecting us: 1. There is still memory hanging around...

> CPU-based dask seems to use much less memory. huh now this also seems to be the opposite. not sure what's up with that.

Sorry, the reshape operation is costly. So ```python import dask.distributed as dd import numpy as np cluster = dd.LocalCluster(n_workers=1) client = dd.Client(cluster) M = 100_000 N = 4_000 def make_chunk():...