[BUG] an error in pySCENIC grnboost2 in Jupyder

[socialtree-yt]

2022-07-01 09:52:42,226 - distributed.worker - WARNING - Heartbeat to scheduler failed Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 229, in read frames_nbytes = await stream.read_bytes(fmt_size) tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/worker.py", line 1151, in heartbeat response = await retry_operation( File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/utils_comm.py", line 383, in retry_operation return await retry( File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/utils_comm.py", line 368, in retry return await coro() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 1158, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 923, in send_recv response = await comm.read(deserializers=deserializers) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 245, in read convert_stream_closed_error(self, e) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 150, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:35894 remote=tcp://127.0.0.1:38538>: Stream is closed 2022-07-01 09:52:42,227 - distributed.worker - WARNING - Heartbeat to scheduler failed Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 229, in read frames_nbytes = await stream.read_bytes(fmt_size) tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/worker.py", line 1151, in heartbeat response = await retry_operation( File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/utils_comm.py", line 383, in retry_operation return await retry( File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/utils_comm.py", line 368, in retry return await coro() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 1158, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 923, in send_recv response = await comm.read(deserializers=deserializers) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 245, in read convert_stream_closed_error(self, e) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 150, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:35906 remote=tcp://127.0.0.1:38538>: Stream is closed 2022-07-01 09:52:42,240 - distributed.worker - WARNING - Heartbeat to scheduler failed Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 229, in read frames_nbytes = await stream.read_bytes(fmt_size) tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/worker.py", line 1151, in heartbeat response = await retry_operation( File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/utils_comm.py", line 383, in retry_operation return await retry( File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/utils_comm.py", line 368, in retry return await coro() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 1158, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 923, in send_recv response = await comm.read(deserializers=deserializers) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 245, in read convert_stream_closed_error(self, e) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 150, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:35958 remote=tcp://127.0.0.1:38538>: Stream is closed 2022-07-01 09:52:42,242 - distributed.worker - WARNING - Heartbeat to scheduler failed Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 229, in read frames_nbytes = await stream.read_bytes(fmt_size) tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/worker.py", line 1151, in heartbeat response = await retry_operation( File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/utils_comm.py", line 383, in retry_operation return await retry( File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/utils_comm.py", line 368, in retry return await coro() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 1158, in send_recv_from_rpc return await send_recv(comm=comm, op=key, **kwargs) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 923, in send_recv response = await comm.read(deserializers=deserializers) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 245, in read convert_stream_closed_error(self, e) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/comm/tcp.py", line 150, in convert_stream_closed_error raise CommClosedError(f"in {obj}: {exc}") from exc distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:35890 remote=tcp://127.0.0.1:38538>: Stream is closed 2022-07-01 09:52:46,160 - distributed.nanny - WARNING - Worker process still alive after 3.999996376037598 seconds, killing 2022-07-01 09:52:46,162 - distributed.nanny - WARNING - Worker process still alive after 3.999998474121094 seconds, killing 2022-07-01 09:52:46,164 - distributed.nanny - WARNING - Worker process still alive after 3.9999973297119142 seconds, killing 2022-07-01 09:52:46,216 - distributed.nanny - WARNING - Worker process still alive after 3.999997901916504 seconds, killing 2022-07-01 09:52:46,218 - distributed.nanny - WARNING - Worker process still alive after 3.999998092651367 seconds, killing 2022-07-01 09:52:46,220 - distributed.nanny - WARNING - Worker process still alive after 3.999998092651367 seconds, killing 2022-07-01 09:52:46,222 - distributed.nanny - WARNING - Worker process still alive after 3.999998092651367 seconds, killing 2022-07-01 09:52:46,224 - distributed.nanny - WARNING - Worker process still alive after 3.999997901916504 seconds, killing 2022-07-01 09:52:46,735 - tornado.application - ERROR - Exception in callback functools.partial(<bound method AsyncProcess._on_exit of <AsyncProcess Dask Worker process (from Nanny)>>, -15) Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/process.py", line 139, in _on_exit self._exit_callback(self) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 695, in _on_exit self.mark_stopped() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 733, in mark_stopped self.on_exit(r) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 501, in _on_exit_sync self._ongoing_background_tasks.call_soon(self._on_exit, exitcode) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 190, in call_soon raise AsyncTaskGroupClosedError( distributed.core.AsyncTaskGroupClosedError: Cannot schedule a new coroutine function as the group is already closed. 2022-07-01 09:52:46,741 - tornado.application - ERROR - Exception in callback functools.partial(<bound method AsyncProcess._on_exit of <AsyncProcess Dask Worker process (from Nanny)>>, -15) Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/process.py", line 139, in _on_exit self._exit_callback(self) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 695, in _on_exit self.mark_stopped() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 733, in mark_stopped self.on_exit(r) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 501, in _on_exit_sync self._ongoing_background_tasks.call_soon(self._on_exit, exitcode) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 190, in call_soon raise AsyncTaskGroupClosedError( distributed.core.AsyncTaskGroupClosedError: Cannot schedule a new coroutine function as the group is already closed. 2022-07-01 09:52:46,744 - tornado.application - ERROR - Exception in callback functools.partial(<bound method AsyncProcess._on_exit of <AsyncProcess Dask Worker process (from Nanny)>>, -15) Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/process.py", line 139, in _on_exit self._exit_callback(self) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 695, in _on_exit self.mark_stopped() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 733, in mark_stopped self.on_exit(r) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 501, in _on_exit_sync self._ongoing_background_tasks.call_soon(self._on_exit, exitcode) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 190, in call_soon raise AsyncTaskGroupClosedError( distributed.core.AsyncTaskGroupClosedError: Cannot schedule a new coroutine function as the group is already closed. 2022-07-01 09:52:46,855 - tornado.application - ERROR - Exception in callback functools.partial(<bound method AsyncProcess._on_exit of <AsyncProcess Dask Worker process (from Nanny)>>, -15) Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/process.py", line 139, in _on_exit self._exit_callback(self) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 695, in _on_exit self.mark_stopped() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 733, in mark_stopped self.on_exit(r) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 501, in _on_exit_sync self._ongoing_background_tasks.call_soon(self._on_exit, exitcode) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 190, in call_soon raise AsyncTaskGroupClosedError( distributed.core.AsyncTaskGroupClosedError: Cannot schedule a new coroutine function as the group is already closed. 2022-07-01 09:52:46,861 - tornado.application - ERROR - Exception in callback functools.partial(<bound method AsyncProcess._on_exit of <AsyncProcess Dask Worker process (from Nanny)>>, -15) Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/process.py", line 139, in _on_exit self._exit_callback(self) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 695, in _on_exit self.mark_stopped() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 733, in mark_stopped self.on_exit(r) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 501, in _on_exit_sync self._ongoing_background_tasks.call_soon(self._on_exit, exitcode) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 190, in call_soon raise AsyncTaskGroupClosedError( distributed.core.AsyncTaskGroupClosedError: Cannot schedule a new coroutine function as the group is already closed. 2022-07-01 09:52:46,907 - tornado.application - ERROR - Exception in callback functools.partial(<bound method AsyncProcess._on_exit of <AsyncProcess Dask Worker process (from Nanny)>>, -15) Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/process.py", line 139, in _on_exit self._exit_callback(self) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 695, in _on_exit self.mark_stopped() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 733, in mark_stopped self.on_exit(r) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 501, in _on_exit_sync self._ongoing_background_tasks.call_soon(self._on_exit, exitcode) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 190, in call_soon raise AsyncTaskGroupClosedError( distributed.core.AsyncTaskGroupClosedError: Cannot schedule a new coroutine function as the group is already closed. 2022-07-01 09:52:46,911 - tornado.application - ERROR - Exception in callback functools.partial(<bound method AsyncProcess._on_exit of <AsyncProcess Dask Worker process (from Nanny)>>, -15) Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/process.py", line 139, in _on_exit self._exit_callback(self) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 695, in _on_exit self.mark_stopped() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 733, in mark_stopped self.on_exit(r) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 501, in _on_exit_sync self._ongoing_background_tasks.call_soon(self._on_exit, exitcode) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 190, in call_soon raise AsyncTaskGroupClosedError( distributed.core.AsyncTaskGroupClosedError: Cannot schedule a new coroutine function as the group is already closed. 2022-07-01 09:52:46,966 - tornado.application - ERROR - Exception in callback functools.partial(<bound method AsyncProcess._on_exit of <AsyncProcess Dask Worker process (from Nanny)>>, -15) Traceback (most recent call last): File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/tornado/ioloop.py", line 741, in _run_callback ret = callback() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/process.py", line 139, in _on_exit self._exit_callback(self) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 695, in _on_exit self.mark_stopped() File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 733, in mark_stopped self.on_exit(r) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/nanny.py", line 501, in _on_exit_sync self._ongoing_background_tasks.call_soon(self._on_exit, exitcode) File "/ifs/home/yanteng/.conda/envs/ryt/lib/python3.10/site-packages/distributed/core.py", line 190, in call_soon raise AsyncTaskGroupClosedError( distributed.core.AsyncTaskGroupClosedError: Cannot schedule a new coroutine function as the group is already closed.

[ghuls]

Try to run with less number of workers, You might have run out of RAM which killed one or more of the python processes.