MP Pebworth

I'd second this. This seems like a critical feature, alongside just getting scArches to work with higher versions of python so it can be used alongside rapids-single cell and other...

Chunks are about 50,000, and the datasets has several million cells. I'll reduce the chunks and try again!

I reduce my chunks to 5,000 and I get the same error. This error does not occur during normalization, so I know it's not The only problem is, I'm running...

Ah... eventually it crashed on CPU, but I think that's because I had memory management on, for GPU memory. return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/mppebworth/miniforge3/envs/newrapids/lib/python3.12/site-packages/distributed/scheduler.py", line 6613, in restart_workers...

Good news: Batch size is not the number to evaluate- it's sample size that matters (Using batch is just faster for small datasets). TLDR: Doublets are only really an issue...

Okay, thank you for trying. Is there no way to identify the optimal size chunks (with an overlap), and then pull in fresh from a zarr store with that batch...