pySCENIC icon indicating copy to clipboard operation
pySCENIC copied to clipboard

Published 20 hours ago • verified publisher icon aertslab

GRN step

Open Sayyam-Shah opened this issue 1 year ago • 4 comments

Hello,

Thank you for the amazing tool.

I am running the GRN step on a huge dataset. It has displayed the below output for a while. Is this normal or should I cancel the job?

preparing dask client distributed.diskutils - INFO - Found stale lock file and directory '/cluster/home/dask-worker-space/worker-iu1orzvb', purging distributed.diskutils - INFO - Found stale lock file and directory '/cluster/home/dask-worker-space/worker-hoff_k94', purging

Sayyam-Shah avatar Jul 16 '22 17:07 Sayyam-Shah

Hello @ghuls,

May you please respond to this? I have run pyscenic multiple times and increased the resources, yet I'm still experiencing the issue. The dataset consists of 270,000 cells. I supplied it with 17 cores and 950 GB.

Sayyam-Shah avatar Aug 16 '22 17:08 Sayyam-Shah

Hi @Sayyam-Shah

To get more info on the warning message please see this answer on stack overflow: https://stackoverflow.com/a/48664628/16278776

That being said, this warning message is not necessarily an issue, but does probably mean that some pysenic run has failed in the past.

Did you get any error messages?

Thanks,

Seppe

SeppeDeWinter avatar Aug 19 '22 07:08 SeppeDeWinter

Hello @SeppeDeWinter,

I'm still able to run pyscenic on smaller datasets so I'm not exactly sure why I'm not experiencing the same issue for them. I ran it for 17 cores and 950 GB on the HPC cluster. The max time allocation is 5 days. At the end of the 5 days, I get the below output.

preparing dask client distributed.diskutils - INFO - Found stale lock file and directory '/cluster/home/t114108uhn/dask-worker-space/worker-35iua5e8', purging distributed.diskutils - INFO - Found stale lock file and directory '/cluster/home/t114108uhn/dask-worker-space/worker-nof12p6u', purging distributed.diskutils - INFO - Found stale lock file and directory '/cluster/home/t114108uhn/dask-worker-space/worker-chx5g2wb', purging distributed.diskutils - INFO - Found stale lock file and directory '/cluster/home/t114108uhn/dask-worker-space/worker-o0k1c0su', purging slurmstepd: error: *** JOB 6227608 ON node96 CANCELLED AT 2022-08-24T16:04:42 DUE TO TIME LIMIT ***

I tried cleaning it up myself by deleting the dask-worker-space directory.

How do I specify where the dask worker places the directory with pyscenic. Maybe, I'll place it in my group directory rather than my home directory.

Sayyam-Shah avatar Aug 26 '22 19:08 Sayyam-Shah

I meet the same problem regardless of the number of cells. Do you find the solution?

singlece avatar Oct 28 '22 06:10 singlece