pySCENIC
pySCENIC copied to clipboard
aertslab
Error running GRNBoost2 step
Error running:
pyscenic grn
--num_workers 20
--output adj.tsv
--method grnboost2
PBMC10k_filtered.loom
hs_hgnc_tfs.txt
(...)
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 269.80 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 263.60 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 267.21 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 268.83 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Heartbeat to scheduler failed
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 268.80 MB -- Worker memory limit: 318.99 MB
distributed.worker - WARNING - Heartbeat to scheduler failed
distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
distributed.nanny - WARNING - Worker process still alive after 3 seconds, killing
(...)
Traceback (most recent call last):
File "/home/echen/miniforge3/envs/scenic/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 675, in main
args.func(args)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/pyscenic/cli/pyscenic.py", line 100, in find_adjacencies_command
expression_data=ex_mtx, tf_names=tf_names, verbose=True, client_or_address=client, seed=args.seed
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/arboreto/algo.py", line 41, in grnboost2
early_stop_window_length=early_stop_window_length, limit=limit, seed=seed, verbose=verbose)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/arboreto/algo.py", line 128, in diy
seed=seed)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/arboreto/core.py", line 421, in create_graph
[future_tf_matrix_gene_names] = client.scatter([tf_matrix_gene_names], broadcast=True)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/client.py", line 2175, in scatter
hash=hash,
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/client.py", line 838, in sync
self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/utils.py", line 351, in sync
raise exc.with_traceback(tb)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/utils.py", line 334, in f
result[0] = yield future
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/tornado/gen.py", line 762, in run
value = future.result()
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/client.py", line 2062, in _scatter
timeout=timeout,
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
result = await send_recv(comm=comm, op=key, **kwargs)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 660, in send_recv
raise exc.with_traceback(tb)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 500, in handle_comm
result = await result
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/scheduler.py", line 5043, in scatter
await self.replicate(keys=keys, workers=workers, n=n)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/scheduler.py", line 5511, in replicate
for w, who_has in gathers.items()
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/utils_comm.py", line 389, in retry_operation
operation=operation,
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/utils_comm.py", line 369, in retry
return await coro()
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 861, in send_recv_from_rpc
result = await send_recv(comm=comm, op=key, **kwargs)
File "/home/echen/miniforge3/envs/scenic/lib/python3.6/site-packages/distributed/core.py", line 662, in send_recv
raise Exception(response["text"])
Exception: 'data'
- pySCENIC version: 0.11.2
- Installation method: pip (followed instructions here -> doi: 10.1038/s41596-020-0336-2)
- OS: WSL Ubuntu
Package list:
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
aiohttp 3.8.6 pypi_0 pypi
aiosignal 1.2.0 pypi_0 pypi
alsa-lib 1.2.7.2 h166bdaf_0 conda-forge
anndata 0.7.8 pypi_0 pypi
arboreto 0.1.6 pypi_0 pypi
arpack 3.7.0 hdefa2d7_2 conda-forge
async-timeout 4.0.2 pypi_0 pypi
asynctest 0.13.0 pypi_0 pypi
attrs 22.2.0 pypi_0 pypi
blosc 1.21.5 h0f2a231_0 conda-forge
bokeh 2.3.3 pypi_0 pypi
boltons 23.1.1 pypi_0 pypi
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.25.0 hd590300_0 conda-forge
ca-certificates 2023.11.17 hbcca054_0 conda-forge
cached-property 1.5.2 pypi_0 pypi
certifi 2021.5.30 py36h5fab9bb_0 conda-forge
cffi 1.14.6 py36hd8eec40_1 conda-forge
charset-normalizer 2.0.12 pypi_0 pypi
click 8.0.4 pypi_0 pypi
cloudpickle 2.2.1 pypi_0 pypi
contextvars 2.4 pypi_0 pypi
ctxcore 0.1.1 pypi_0 pypi
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.11.0 py36h7b6447c_0 anaconda
dask 2021.3.0 pypi_0 pypi
dbus 1.13.6 h5008d03_3 conda-forge
decorator 4.4.2 pypi_0 pypi
dill 0.3.4 pypi_0 pypi
distributed 2021.3.0 pypi_0 pypi
expat 2.5.0 hcb278e6_1 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 h77eed37_1 conda-forge
fontconfig 2.14.2 h14ed4e7_0 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
freetype 2.12.1 h267a509_2 conda-forge
frozendict 2.4.0 pypi_0 pypi
frozenlist 1.2.0 pypi_0 pypi
fsspec 2022.1.0 pypi_0 pypi
get-version 2.1 pypi_0 pypi
gettext 0.21.1 h27087fc_0 conda-forge
glib 2.78.3 hfc55251_0 conda-forge
glib-tools 2.78.3 hfc55251_0 conda-forge
glpk 4.65 h9202a9a_1004 conda-forge
gmp 6.3.0 h59595ed_0 conda-forge
gst-plugins-base 1.20.3 h57caac4_2 conda-forge
gstreamer 1.20.3 hd4edc92_2 conda-forge
h5py 3.1.0 pypi_0 pypi
hdf5 1.12.1 nompi_h2386368_104 conda-forge
heapdict 1.0.1 pypi_0 pypi
icu 69.1 h9c3ff4c_0 conda-forge
idna 3.6 pypi_0 pypi
idna-ssl 1.1.0 pypi_0 pypi
immutables 0.19 pypi_0 pypi
importlib-metadata 4.8.3 pypi_0 pypi
importlib-resources 5.4.0 pypi_0 pypi
interlap 0.2.7 pypi_0 pypi
jinja2 3.0.3 pypi_0 pypi
joblib 1.2.0 pyhd8ed1ab_0 conda-forge
jpeg 9e h0b41bf4_3 conda-forge
keyutils 1.6.1 h166bdaf_0 conda-forge
kiwisolver 1.3.1 py36h605e78d_1 conda-forge
krb5 1.20.1 hf9c8cef_0 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.40 h41732ed_0 conda-forge
legacy-api-wrap 1.2 pypi_0 pypi
lerc 3.0 h9c3ff4c_0 conda-forge
libblas 3.9.0 20_linux64_openblas conda-forge
libcblas 3.9.0 20_linux64_openblas conda-forge
libclang 13.0.1 default_h7634d5b_5 conda-forge
libcurl 7.87.0 h6312ad2_0 conda-forge
libdeflate 1.10 h7f98852_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 hd590300_2 conda-forge
libevent 2.1.10 h9b69904_4 conda-forge
libexpat 2.5.0 hcb278e6_1 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h807b86a_3 conda-forge
libgfortran-ng 13.2.0 h69a702a_3 conda-forge
libgfortran5 13.2.0 ha4646dd_3 conda-forge
libglib 2.78.3 h783c2da_0 conda-forge
libgomp 13.2.0 h807b86a_3 conda-forge
libiconv 1.17 hd590300_2 conda-forge
liblapack 3.9.0 20_linux64_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libllvm13 13.0.1 hf817b99_2 conda-forge
libnghttp2 1.51.0 hdcd2b5c_0 conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.25 pthreads_h413a1c8_0 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.39 h753d276_0 conda-forge
libpq 14.5 h2baec63_5 conda-forge
libsqlite 3.44.2 h2797004_0 conda-forge
libssh2 1.10.0 haa6b8db_3 conda-forge
libstdcxx-ng 13.2.0 h7e041cc_3 conda-forge
libtiff 4.3.0 h0fcbabc_4 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.3.2 hd590300_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.14 haae042b_4 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
llvmlite 0.36.0 py36h05121d2_0 conda-forge
locket 1.0.0 pypi_0 pypi
loompy 3.0.7 pypi_0 pypi
louvain 0.7.0 py36hf105032_1 conda-forge
lz4-c 1.9.4 hcb278e6_0 conda-forge
lzo 2.10 h516909a_1000 conda-forge
markupsafe 2.0.1 pypi_0 pypi
matplotlib 3.3.4 py36h5fab9bb_0 conda-forge
matplotlib-base 3.3.4 py36hd391965_0 conda-forge
metis 5.1.0 h59595ed_1007 conda-forge
mock 5.1.0 pyhd8ed1ab_0 conda-forge
mpfr 4.2.1 h9458935_0 conda-forge
msgpack 1.0.5 pypi_0 pypi
multicore-tsne 0.1_d4ff4aab py36h706143c_1 conda-forge
multidict 5.2.0 pypi_0 pypi
multiprocessing-on-dill 3.5.0a4 pypi_0 pypi
mysql-common 8.0.32 h14678bc_0 conda-forge
mysql-libs 8.0.32 h54cf53e_0 conda-forge
natsort 8.2.0 pypi_0 pypi
ncurses 6.4 h59595ed_2 conda-forge
networkx 2.5.1 pypi_0 pypi
nspr 4.35 h27087fc_0 conda-forge
nss 3.96 h1d7d5a4_0 conda-forge
numba 0.53.1 py36h1328434_1 conda-forge
numexpr 2.7.3 py36h0cdc3f0_0 conda-forge
numpy 1.19.5 py36hfc0c790_2 conda-forge
numpy-groupies 0.9.22 pypi_0 pypi
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjpeg 2.5.0 h7d73246_0 conda-forge
openssl 1.1.1w hd590300_0 conda-forge
packaging 21.3 pypi_0 pypi
pandas 1.1.5 py36h284efc9_0 conda-forge
partd 1.2.0 pypi_0 pypi
patsy 0.5.6 pyhd8ed1ab_0 conda-forge
pcre2 10.42 hcad00b1_0 conda-forge
pillow 8.3.2 py36h676a545_0 conda-forge
pip 21.3.1 pyhd8ed1ab_0 conda-forge
psutil 5.9.7 pypi_0 pypi
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pyarrow 0.16.0 pypi_0 pypi
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pynndescent 0.5.11 pypi_0 pypi
pyparsing 3.1.1 pyhd8ed1ab_0 conda-forge
pyqt 5.12.3 py36h5fab9bb_7 conda-forge
pyqt-impl 5.12.3 py36h7ec31b9_7 conda-forge
pyqt5-sip 4.19.18 py36hc4f0c31_7 conda-forge
pyqtchart 5.12 py36h7ec31b9_7 conda-forge
pyqtwebengine 5.12.1 py36h7ec31b9_7 conda-forge
pyscenic 0.11.2 pypi_0 pypi
pytables 3.6.1 py36he0b5dfd_4 conda-forge
python 3.6.15 hb7a2778_0_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-igraph 0.9.6 py36h644ed5e_0 conda-forge
python_abi 3.6 2_cp36m conda-forge
pytz 2023.3.post1 pyhd8ed1ab_0 conda-forge
pyyaml 6.0.1 pypi_0 pypi
qt 5.12.9 h1304e3e_6 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.27.1 pypi_0 pypi
scanpy 1.7.2 pypi_0 pypi
scikit-learn 0.24.2 py36hc89565f_1 conda-forge
scipy 1.5.3 py36h81d768a_1 conda-forge
seaborn 0.11.2 hd8ed1ab_0 conda-forge
seaborn-base 0.11.2 pyhd8ed1ab_0 conda-forge
setuptools 58.0.4 py36h5fab9bb_2 conda-forge
sinfo 0.3.4 pypi_0 pypi
six 1.16.0 pyh6c4a22f_0 conda-forge
snappy 1.1.10 h9fff704_0 conda-forge
sortedcontainers 2.4.0 pypi_0 pypi
sqlite 3.44.2 h2c6b66d_0 conda-forge
statsmodels 0.12.2 py36he33b4a0_0 conda-forge
stdlib-list 0.8.0 pypi_0 pypi
suitesparse 5.10.1 hd8046ac_0 conda-forge
tbb 2020.2 h4bd325d_4 conda-forge
tblib 1.7.0 pypi_0 pypi
texttable 1.7.0 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.1.0 pyh8a188c0_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
toolz 0.11.2 pyhd3eb1b0_0 anaconda
tornado 6.1 py36h8f6f2f9_1 conda-forge
tqdm 4.64.1 pypi_0 pypi
typing-extensions 4.1.1 pypi_0 pypi
umap-learn 0.5.5 pypi_0 pypi
urllib3 1.26.18 pypi_0 pypi
wheel 0.37.1 pyhd8ed1ab_0 conda-forge
xlrd 1.2.0 pypi_0 pypi
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yarl 1.7.2 pypi_0 pypi
zict 2.1.0 pypi_0 pypi
zipp 3.6.0 pypi_0 pypi
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.5 hfc55251_0 conda-forge
Tried with 10 workers, this was the end of the error message:
OSError: Timed out during handshake while connecting to tcp://127.0.0.1:45347 after 10 s
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 446.74 MB -- Worker memory limit: 637.97 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 446.74 MB -- Worker memory limit: 637.97 MB
distributed.worker - WARNING - Memory use is high but worker has no data to store to disk. Perhaps some other process is leaking memory? Process memory: 446.74 MB -- Worker memory limit: 637.97 MB
Killed
(scenic) echen@20766MOPLT:~/working/scenic$ /home/echen/miniforge3/envs/scenic/lib/python3.6/multiprocessing/semaphore_tracker.py:143: UserWarning: semaphore_tracker: There appear to be 48 leaked semaphores to clean up at shutdown
len(cache))
How much RAM do your have? You might need to lower the number of threads even more. Also note that we only run pySCENIC on servers and not on local computers (which might not have enough RAM).
Hi, I also encountered a problem when running GRNBoost2 step:
pyscenic grn --num_workers 20
--seed 42
--method grnboost2
--output ./results/grn_def.csv
sndef.csv allTFs_mm.txt
2024-03-22 10:36:57,207 - pyscenic.cli.pyscenic - INFO - Loading expression matrix.
2024-03-22 10:38:27,167 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks.
Traceback (most recent call last):
File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/bin/pyscenic", line 8, in <module>
sys.exit(main())
File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 713, in main
args.func(args)
File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/pyscenic/cli/pyscenic.py", line 106, in find_adjacencies_command
network = method(
File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/arboreto/algo.py", line 39, in grnboost2
return diy(expression_data=expression_data, regressor_type='GBM', regressor_kwargs=SGBM_KWARGS,
File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/arboreto/algo.py", line 120, in diy
graph = create_graph(expression_matrix,
File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/arboreto/core.py", line 450, in create_graph
all_meta_df = from_delayed(delayed_meta_dfs, meta=_META_SCHEMA)
File "/home/wsq_pkuhpc/lustre2/user/hy/miniconda3/envs/pyscenic/lib/python3.10/site-packages/dask_expr/io/_delayed.py", line 93, in from_delayed
raise TypeError("Must supply at least one delayed object")
TypeError: Must supply at least one delayed object
I am using pySCENIC 0.12.1+2.geaf23eb on a 24 core 1000G RAM HPC
package list
# Name Version Build Channel
_libgcc_mutex 0.1 main defaults
_openmp_mutex 5.1 1_gnu defaults
aiohttp 3.9.3 pypi_0 pypi
aiosignal 1.3.1 pypi_0 pypi
anndata 0.10.6 pypi_0 pypi
arboreto 0.1.6 pypi_0 pypi
array-api-compat 1.5.1 pypi_0 pypi
asttokens 2.4.1 pypi_0 pypi
async-timeout 4.0.3 pypi_0 pypi
attrs 23.2.0 pypi_0 pypi
bokeh 3.4.0 pypi_0 pypi
boltons 23.1.1 pypi_0 pypi
bzip2 1.0.8 h5eee18b_5 defaults
ca-certificates 2024.3.11 h06a4308_0 defaults
certifi 2024.2.2 pypi_0 pypi
charset-normalizer 3.3.2 pypi_0 pypi
click 8.1.7 pypi_0 pypi
cloudpickle 3.0.0 pypi_0 pypi
comm 0.2.2 pypi_0 pypi
contourpy 1.2.0 pypi_0 pypi
ctxcore 0.2.0 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
cytoolz 0.12.3 pypi_0 pypi
dask 2024.3.1 pypi_0 pypi
dask-expr 1.0.4 pypi_0 pypi
debugpy 1.8.1 pypi_0 pypi
decorator 5.1.1 pypi_0 pypi
dill 0.3.8 pypi_0 pypi
distributed 2024.3.1 pypi_0 pypi
exceptiongroup 1.2.0 pypi_0 pypi
executing 2.0.1 pypi_0 pypi
fonttools 4.50.0 pypi_0 pypi
frozendict 2.4.0 pypi_0 pypi
frozenlist 1.4.1 pypi_0 pypi
fsspec 2024.3.1 pypi_0 pypi
h5py 3.10.0 pypi_0 pypi
idna 3.6 pypi_0 pypi
importlib-metadata 7.1.0 pypi_0 pypi
interlap 0.2.7 pypi_0 pypi
ipykernel 6.29.3 pypi_0 pypi
ipython 8.22.2 pypi_0 pypi
jedi 0.19.1 pypi_0 pypi
jinja2 3.1.3 pypi_0 pypi
joblib 1.3.2 pypi_0 pypi
jupyter-client 8.6.1 pypi_0 pypi
jupyter-core 5.7.2 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
ld_impl_linux-64 2.38 h1181459_1 defaults
libffi 3.4.4 h6a678d5_0 defaults
libgcc-ng 11.2.0 h1234567_1 defaults
libgomp 11.2.0 h1234567_1 defaults
libstdcxx-ng 11.2.0 h1234567_1 defaults
libuuid 1.41.5 h5eee18b_0 defaults
llvmlite 0.42.0 pypi_0 pypi
locket 1.0.0 pypi_0 pypi
loompy 3.0.7 pypi_0 pypi
lz4 4.3.3 pypi_0 pypi
markupsafe 2.1.5 pypi_0 pypi
matplotlib 3.8.3 pypi_0 pypi
matplotlib-inline 0.1.6 pypi_0 pypi
msgpack 1.0.8 pypi_0 pypi
multidict 6.0.5 pypi_0 pypi
multiprocessing-on-dill 3.5.0a4 pypi_0 pypi
natsort 8.4.0 pypi_0 pypi
ncurses 6.4 h6a678d5_0 defaults
nest-asyncio 1.6.0 pypi_0 pypi
networkx 3.2.1 pypi_0 pypi
numba 0.59.1 pypi_0 pypi
numexpr 2.9.0 pypi_0 pypi
numpy 1.26.4 pypi_0 pypi
numpy-groupies 0.10.2 pypi_0 pypi
openssl 3.0.13 h7f8727e_0 defaults
packaging 24.0 pypi_0 pypi
pandas 2.2.1 pypi_0 pypi
parso 0.8.3 pypi_0 pypi
partd 1.4.1 pypi_0 pypi
patsy 0.5.6 pypi_0 pypi
pexpect 4.9.0 pypi_0 pypi
pillow 10.2.0 pypi_0 pypi
pip 23.3.1 pypi_0 pypi
platformdirs 4.2.0 pypi_0 pypi
prompt-toolkit 3.0.43 pypi_0 pypi
psutil 5.9.8 pypi_0 pypi
ptyprocess 0.7.0 pypi_0 pypi
pure-eval 0.2.2 pypi_0 pypi
pyarrow 15.0.2 pypi_0 pypi
pyarrow-hotfix 0.6 pypi_0 pypi
pygments 2.17.2 pypi_0 pypi
pynndescent 0.5.11 pypi_0 pypi
pyparsing 3.1.2 pypi_0 pypi
pyscenic 0.12.1+2.geaf23eb pypi_0 pypi
python 3.10.13 h955ad1f_0 defaults
python-dateutil 2.9.0.post0 pypi_0 pypi
pytz 2024.1 pypi_0 pypi
pyyaml 6.0.1 pypi_0 pypi
pyzmq 25.1.2 pypi_0 pypi
readline 8.2 h5eee18b_0 defaults
requests 2.31.0 pypi_0 pypi
scanpy 1.9.8 pypi_0 pypi
scikit-learn 1.4.1.post1 pypi_0 pypi
scipy 1.12.0 pypi_0 pypi
seaborn 0.13.2 pypi_0 pypi
session-info 1.0.0 pypi_0 pypi
setuptools 68.2.2 pypi_0 pypi
six 1.16.0 pypi_0 pypi
sortedcontainers 2.4.0 pypi_0 pypi
sqlite 3.41.2 h5eee18b_0 defaults
stack-data 0.6.3 pypi_0 pypi
statsmodels 0.14.1 pypi_0 pypi
stdlib-list 0.10.0 pypi_0 pypi
tblib 3.0.0 pypi_0 pypi
threadpoolctl 3.4.0 pypi_0 pypi
tk 8.6.12 h1ccaba5_0 defaults
toolz 0.12.1 pypi_0 pypi
tornado 6.4 pypi_0 pypi
tqdm 4.66.2 pypi_0 pypi
traitlets 5.14.2 pypi_0 pypi
tzdata 2024.1 pypi_0 pypi
umap-learn 0.5.5 pypi_0 pypi
urllib3 2.2.1 pypi_0 pypi
wcwidth 0.2.13 pypi_0 pypi
wheel 0.41.2 pypi_0 pypi
xyzservices 2023.10.1 pypi_0 pypi
xz 5.4.6 h5eee18b_0 defaults
yarl 1.9.4 pypi_0 pypi
zict 3.0.0 pypi_0 pypi
zipp 3.18.1 pypi_0 pypi
zlib 1.2.13 h5eee18b_0 defaults
I met the same problem. I guessed it was probably due to version incompatibility. So I changed the version of packages. However, it didn't work. And when I input as follows, it ran out results.
pyscenic ctx --num_workers 10
--output regulons.csv
--expression_mtx_fname sce.loom
--mode "custom_multiprocessing"
--annotations_fname motifs-v9-nr.hgnc-m0.001-o0.0.tbl
sce.adj.csv
hg38_10kbp_up_10kbp_down_full_tx_v10_clust.genes_vs_motifs.rankings.feather
Ocean-Lyu I met the same problem as you. But I did't find a resolution.
Also the same problem. It seems that this issue has only arisen recently.
Changing dask/distributed version worked for me.
You can try pip install distributed==2023.12.1 !
If you got error message of "TypeError: Must supply at least one delayed object", It may be caused by dependency error of dask-expr package. dask-expr package was currently updated from version 0.5.3 (Feb 28) to 1.0 (Mar 12)
You could avoid this message by down-grading dask-expr package, by following code
pip install dask-expr==0.5.3
git version of arboreto should fix this: https://github.com/aertslab/arboreto/pull/39
If you got error message of "TypeError: Must supply at least one delayed object", It may be caused by dependency error of dask-expr package. dask-expr package was currently updated from version 0.5.3 (Feb 28) to 1.0 (Mar 12)
You could avoid this message by down-grading dask-expr package, by following code
pip install dask-expr==0.5.3
Merci bcp!!! C'etait Vrai!!!
Hi guys, I used HPC without Root access. and I use conda environment and command line of pyscenic. I solve this problems by @r-trimbour and @dactylogram their answer: pip install dask-expr==0.5.3 distributed==2024.2.1 only install on package will cause "ERROR: pip's dependency resolver does not currently take into account all..."
Hi guys, I used HPC without Root access. and I use conda environment and command line of pyscenic. I solve this problems by @r-trimbour and @dactylogram their answer: pip install dask-expr==0.5.3 distributed==2024.2.1 only install on package will cause "ERROR: pip's dependency resolver does not currently take into account all..." oui, mais je sais pas pourquoi a cette question
Hi @yifanfu01 I don't know if you're still struggling with the installation, but if it's the case I think dask depends on distributed so you have to choose the same version for both of them. We were both suggesting to update their versions but you can choose as long as it's before marche 2024 on pip from what I remember. :)
git version of arboreto should fix this: aertslab/arboreto#39
running into this issue, https://github.com/aertslab/arboreto/issues/33, when trying to install arboreto from source..
I have a similar problem plus "ERROR - Error in Nanny killing Worker subprocess". I can't understand if it's because I put "--rm" as the argument. It automatically removes the container once it exists. Moreover, the problem was apparently resolved below by the explanation "The above exception was the direct cause of the following exception". Help me understand the error so I can figure out how I need to run the analysis again.
I use 150GB and 20 CPU
docker run -it --rm \
-v /data:/data \
aertslab/pyscenic:0.12.1 pyscenic grn \
--num_workers 20 \
-o /data/expr_mat.adjacencies.tsv \
/data/expr_mat.loom \
/data/allTFs_hg38.txt
024-06-25 21:09:47,005 - pyscenic.cli.pyscenic - INFO - Inferring regulatory networks. 2024-06-26 16:12:16,777 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:16,778 - distributed.nanny - WARNING - Worker process still alive after 3.1999989318847657 seconds, killing 2024-06-26 16:12:16,778 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,778 - distributed.nanny - WARNING - Worker process still alive after 3.1999995422363288 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2024-06-26 16:12:16,779 - distributed.nanny - WARNING - Worker process still alive after 3.1999986267089846 seconds, killing 2024-06-26 16:12:16,780 - distributed.nanny - WARNING - Worker process still alive after 3.1999990844726565 seconds, killing 2024-06-26 16:12:16,780 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:16,780 - distributed.nanny - WARNING - Worker process still alive after 3.1999989318847657 seconds, killing 2024-06-26 16:12:16,782 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:16,782 - distributed.nanny - WARNING - Worker process still alive after 3.199998474121094 seconds, killing 2024-06-26 16:12:16,782 - distributed.nanny - WARNING - Worker process still alive after 3.199999389648438 seconds, killing 2024-06-26 16:12:16,782 - distributed.nanny - WARNING - Worker process still alive after 3.1999992370605472 seconds, killing 2024-06-26 16:12:17,493 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,495 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,496 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,498 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,499 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,503 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,542 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,545 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,568 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError 2024-06-26 16:12:17,570 - distributed.nanny - ERROR - Error in Nanny killing Worker subprocess Traceback (most recent call last): File "/usr/local/lib/python3.10/asyncio/tasks.py", line 456, in wait_for return fut.result() asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last): File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 580, in close await self.kill(timeout=timeout) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 372, in kill await self.process.kill(timeout=0.8 * (deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/nanny.py", line 781, in kill await process.join(max(0, deadline - time())) File "/opt/venv/lib/python3.10/site-packages/distributed/process.py", line 304, in join await asyncio.wait_for(asyncio.shield(self._exit_future), timeout) File "/usr/local/lib/python3.10/asyncio/tasks.py", line 458, in wait_for raise exceptions.TimeoutError() from exc asyncio.exceptions.TimeoutError
2024-06-26 16:12:17,576 - pyscenic.cli.pyscenic - INFO - Writing results to file. preparing dask client parsing input creating dask graph 20 partitions computing dask graph not shutting down client, client was created externally finished
