grnboost error

[wangjiawen2013]

Hello, When I run "network = grnboost2(expression_data=ex_matrix,tf_names=tf_names)" according to the tutorial, the following error occured: here is a snippet of the errors:

distributed.core - ERROR - Timed out trying to connect to 'tcp://127.0.0.1:33261' after 10 s: connect() didn't finish in time tornado.util.TimeoutError: Timeout OSError: Timed out trying to connect to 'tcp://127.0.0.1:33261' after 10 s: connect() didn't finish in time tornado.application - ERROR - Exception in Future <Future cancelled> after timeout distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:56994 remote=tcp://127.0.0.1:56550> distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:56995 remote=tcp://127.0.0.1:56550> distributed.comm.tcp - WARNING - Closing dangling stream in <TCP local=tcp://127.0.0.1:56997 remote=tcp://127.0.0.1:56550>

[tmoerman]

Based on your pySCENIC issue, I think the Tornado error is not the root cause. Tornado probably times out because of a problem with the configuration of the numerical libraries on your system, OpenBLAS in this case.

Perhaps following StackOverflow question can help you resolve the matter: https://stackoverflow.com/questions/51256738/multiple-instances-of-python-running-simultaneously-limited-to-35

PS: it really helps posting issues with more contextual information, like the other issue you filed, longer snippet from the error log, information on what kind of system you are running (operating system, specs etc.), and what Python environment you are using.

[wangjiawen2013]

I still cannot run grnboost although I change OpenBLAS settings according to your indication, here are the detailed error message:

tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start tornado.application - ERROR - Multiple exceptions in yield list Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start Traceback (most recent call last): File "pipeline", line 18, in adjacencies = grnboost2(ex_matrix, tf_names=tf_names, verbose=True) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/arboreto/algo.py", line 41, in grnboost2 early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/arboreto/algo.py", line 109, in diy client, shutdown_callback = _prepare_client(client_or_address) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/arboreto/algo.py", line 157, in _prepare_client local_cluster = LocalCluster(diagnostics_port=None) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 141, in init self.start(ip=ip, n_workers=n_workers) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 171, in start self.sync(self._start, **kwargs) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 164, in sync return sync(self.loop, func, *args, **kwargs) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/utils.py", line 277, in sync six.reraise(*error[0]) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/six.py", line 693, in reraise raise value File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/utils.py", line 262, in f result[0] = yield future File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run yielded = self.gen.throw(*exc_info) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 191, in _start yield [self._start_worker(**self.worker_kwargs) for i in range(n_workers)] File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run value = future.result() File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 883, in callback result_list.append(f.result()) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/tornado/gen.py", line 1147, in run yielded = self.gen.send(value) File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 217, in _start_worker raise gen.TimeoutError("Worker failed to start") tornado.util.TimeoutError: Worker failed to start distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-35, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-56, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-62, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-13, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-2, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-38, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-27, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-16, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-41, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-5, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-30, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-60, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-19, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-8, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-44, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-47, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-11, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-14, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-1, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-3, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-53, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-17, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-42, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-6, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-31, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-20, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-59, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-63, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-23, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-48, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-15, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-40, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-54, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-18, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-7, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-64, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-21, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-10, started daemon)> distributed.process - WARNING - reaping stray process <ForkServerProcess(ForkServerProcess-46, started daemon)> Exception ignored in: <bound method LocalCluster.del of LocalCluster('tcp://127.0.0.1:40281', workers=0, ncores=0)> Traceback (most recent call last): File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 340, in del File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/deploy/local.py", line 291, in close File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/utils.py", line 425, in run_sync File "/home/wangjw/miniconda3/envs/python36/lib/python3.6/site-packages/distributed/utils.py", line 272, in sync tornado.util.TimeoutError: timed out after 20 s.

[vagabond12]

@wangjiawen2013 I have the same problem, Have you solve this problem? Thank you.

[wangjiawen2013]

@vagabond12 solve it like this: arboreto_with_multiprocessing.py
expr_mat.loom
allTFs_hg38.txt
--method grnboost2
--output adj.tsv
--num_workers 20
--seed 777 https://pyscenic.readthedocs.io/en/latest/faq.html

the latest pyscenic added a helper script arboreto_with_multiprocessing.py that runs the Arboreto GRN algorithms (GRNBoost2, GENIE3) without Dask for compatibility. Then you can import the results into jupyter notebook.

[vagabond12]

Thank you, I know this helper script. But I do not think this helper script runs the Arboreto GRN algorithms (GRNBoost2, GENIE3) without Dask for compatibility, Because it has the same warning of dask as pyscenic grn. If they all use dask, Why pyscenic grn always does not work? That is a question.

[wangjiawen2013]

I think the warnings are irrevelant to dask. It's just on the configuration parameters (in .yaml format) or other things.