Errors with running pySCENIC in conda environment or through docker/singularity images

[fouerghi]

I am having issues running pySCENIC. I have attempted to set up a conda environment to use it, as well as using the Docker and Singularity images. I am still getting the following errors:

2021-08-05 17:04:15,761 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.

2021-08-05 17:04:24,956 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks. distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.scheduler - ERROR - Couldn't gather keys {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ['tcp://127.0.0.1:41234']} state: ['waiting'] workers: ['tcp://127.0.0.1:41234'] NoneType: None distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://127.0.0.1:41234'], finalize-dc3dc6e36a9e8076a3b034d8407946e5 NoneType: None distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ('tcp://127.0.0.1:41234',)} distributed.nanny - WARNING - Restarting worker preparing dask client parsing input creating dask graph 20 partitions computing dask graph distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.scheduler - ERROR - Couldn't gather keys {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ['tcp://127.0.0.1:46797']} state: ['waiting'] workers: ['tcp://127.0.0.1:46797'] NoneType: None distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://127.0.0.1:46797'], finalize-dc3dc6e36a9e8076a3b034d8407946e5 NoneType: None distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ('tcp://127.0.0.1:46797',)} distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.scheduler - ERROR - Couldn't gather keys {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ['tcp://127.0.0.1:45718']} state: ['waiting'] workers: ['tcp://127.0.0.1:45718'] NoneType: None distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://127.0.0.1:45718'], finalize-dc3dc6e36a9e8076a3b034d8407946e5 NoneType: None distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ('tcp://127.0.0.1:45718',)} distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.scheduler - ERROR - Couldn't gather keys {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ['tcp://127.0.0.1:33865']} state: ['waiting'] workers: ['tcp://127.0.0.1:33865'] NoneType: None distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://127.0.0.1:33865'], finalize-dc3dc6e36a9e8076a3b034d8407946e5 NoneType: None distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ('tcp://127.0.0.1:33865',)} distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.nanny - WARNING - Restarting worker distributed.nanny - WARNING - Worker exceeded 95% memory budget. Restarting distributed.scheduler - ERROR - Couldn't gather keys {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ['tcp://127.0.0.1:43295']} state: ['waiting'] workers: ['tcp://127.0.0.1:43295'] NoneType: None distributed.scheduler - ERROR - Workers don't have promised key: ['tcp://127.0.0.1:43295'], finalize-dc3dc6e36a9e8076a3b034d8407946e5 NoneType: None distributed.client - WARNING - Couldn't gather 1 keys, rescheduling {'finalize-dc3dc6e36a9e8076a3b034d8407946e5': ('tcp://127.0.0.1:43295',)} distributed.nanny - WARNING - Restarting worker

Any help with solving this?

[Yippee3]

hello, have you solved it? please. I just had this problem too

[Famingzhao]

Me too. It is a strange bug.

[ishwarvh]

Is there a solution to this issue? I am also seeing the same error messages despite allocating fewer workers and increasing memory allocation. I tried pySCENIC because it was said it is "lightning" fast compare to the R version. But the process is stuck at the GRN step for 4 days!

[Yippee3]

Is there a solution to this issue? I am also seeing the same error messages despite allocating fewer workers and increasing memory allocation. I tried pySCENIC because it was said it is "lightning" fast compare to the R version. But the process is stuck at the GRN step for 4 days!

When I tried 10w cells it took me seven days(36 cores and 500G memory). And try random sampling.