Kernel dies with 500GB memory

[kotr98]

Hi,

i am trying to integrate 19 scanpy adata objects with a total of 65k cells and 2000 (hvg) genes. I just want to get the embeddings. I used the code snipped from the scib-metrics tutorial (https://scib-metrics.readthedocs.io/en/stable/notebooks/lung_example.html):

# List of adata per sample
batch_cats = adata_final2.obs["sample"].cat.categories
adata_list = [adata_final2[adata_final2.obs["sample"] == b].copy() for b in batch_cats]

scanorama.integrate_scanpy(adata_list)

adata_final2.obsm["Scanorama"] = np.zeros((adata_final2.shape[0], adata_list[0].obsm["X_scanorama"].shape[1]))
for i, b in enumerate(batch_cats):
    adata_final2.obsm["Scanorama"][adata_final2.obs["sample"] == b] = adata_list[i].obsm["X_scanorama"]

The problem now is, that everytime i try to run it, my kernel dies. I tried to increase memory to 500GB, and it still dies. What are my options?

[brianhie]

Can you try using the sketch parameter to reduce memory consumption? Maybe set the sketch value to a low enough value to fit in memory, if you are indeed getting an OOM error

[kotr98]

I tried using the sketch parameter and it still died. In my slurm error logs i don't get an oom error but just an 709329 Illegal instruction (core dumped) error message.

So it's probably not an oom issue, but i still dont know what the problem is.