aertslab
SCENIC+ crashed in tf_to_gene step
Dear SCENIC+ team,
I am using SCENIC+ and it crashed in tf_to_gene step. The error log is as follows:
Assuming unrestricted shared filesystem usage for local execution.
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 4
Rules claiming more threads will be scaled down.
Job stats:
job count
--------------- -------
AUCell_direct 1
AUCell_extended 1
all 1
eGRN_direct 1
eGRN_extended 1
scplus_mudata 1
tf_to_gene 1
total 7
Select jobs to execute...
Execute 1 jobs...
[Wed Aug 21 13:30:19 2024]
localrule tf_to_gene:
input: /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu, /home/jkang5/sc163/scenicplus/outs/tf_names.txt
output: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv
jobid: 5
reason: Missing output files: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv
threads: 4
resources: tmpdir=/tmp
2024-08-21 13:31:00,298 SCENIC+ INFO Reading multiome MuData.
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/anndata/_core/anndata.py:522: FutureWarning: The dtype argument is deprecated and will be removed in late 2024.
warnings.warn(
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/anndata/_core/anndata.py:522: FutureWarning: The dtype argument is deprecated and will be removed in late 2024.
warnings.warn(
2024-08-21 13:31:36,668 SCENIC+ INFO Using 747 TFs.
2024-08-21 13:31:37,073 TF2G INFO Calculating TF-to-gene importance
Running using 4 cores: 0%| | 0/27636 [00:00<?, ?it/s]
Running using 4 cores: 0%| | 4/27636 [00:00<40:53, 11.26it/s]
Running using 4 cores: 0%| | 4/27636 [00:19<40:53, 11.26it/s]
...
Running using 4 cores: 100%|█████████▉| 27632/27636 [62:57:00<00:21, 5.38s/it]
Running using 4 cores: 100%|██████████| 27636/27636 [62:57:20<00:00, 5.30s/it]
Running using 4 cores: 100%|██████████| 27636/27636 [62:57:20<00:00, 8.20s/it]
2024-08-24 04:29:27,730 TF2G INFO Adding correlation coefficients to adjacencies.
[Sat Aug 24 04:29:41 2024]
Error in rule tf_to_gene:
jobid: 5
input: /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu, /home/jkang5/sc163/scenicplus/outs/tf_names.txt
output: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv
shell:
scenicplus grn_inference TF_to_gene --multiome_mudata_fname /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu --tf_names /home/jkang5/sc163/scenicplus/outs/tf_names.txt --temp_dir /home/jkang5/sc163/scenicplus/temp --out_tf_to_gene_adjacencies /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv --method GBM --n_cpu 4 --seed 666
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
Complete log: .snakemake/log/2024-08-21T133013.885952.snakemake.log
WorkflowError:
At least one job did not complete successfully.
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/joblib/externals/loky/backend/resource_tracker.py:314: UserWarning: resource_tracker: There appear to be 10 leaked semlock objects to clean up at shutdown
warnings.warn(
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/joblib/externals/loky/backend/resource_tracker.py:314: UserWarning: resource_tracker: There appear to be 1 leaked folder objects to clean up at shutdown
warnings.warn(
/rsrch5/home/sct-rsch/jkang5/venvs/scenicplus/lib/python3.11/site-packages/joblib/externals/loky/backend/resource_tracker.py:330: UserWarning: resource_tracker: /home/jkang5/sc163/scenicplus/temp/joblib_memmapping_folder_3191096_65ae4f6fde3149ff8e587d14176cb8c4_e980a3c3e49b4ce9bf7e2b107c50d27a: FileNotFoundError(2, 'No such file or directory')
warnings.warn(f"resource_tracker: {name}: {e!r}")
Could you kindly help resolve this issue?
Thanks, Jack
Hi @kangjiajinlong
Could you share the log file? .snakemake/log/2024-08-21T133013.885952.snakemake.log
All the best,
Seppe
This is the full content of that file:
Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 4 Rules claiming more threads will be scaled down. Job stats: job count
AUCell_direct 1 AUCell_extended 1 all 1 eGRN_direct 1 eGRN_extended 1 scplus_mudata 1 tf_to_gene 1 total 7
Select jobs to execute... Execute 1 jobs...
[Wed Aug 21 13:30:19 2024] localrule tf_to_gene: input: /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu, /home/jkang5/sc163/scenicplus/outs/tf_names.txt output: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv jobid: 5 reason: Missing output files: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv threads: 4 resources: tmpdir=/tmp
[Sat Aug 24 04:29:41 2024] Error in rule tf_to_gene: jobid: 5 input: /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu, /home/jkang5/sc163/scenicplus/outs/tf_names.txt output: /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv shell:
scenicplus grn_inference TF_to_gene --multiome_mudata_fname /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu --tf_names /home/jkang5/sc163/scenicplus/outs/tf_names.txt --temp_dir /home/jkang5/sc163/scenicplus/temp --out_tf_to_gene_adjacencies /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv --method GBM --n_cpu 4 --seed 666
(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)
Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: .snakemake/log/2024-08-21T133013.885952.snakemake.log WorkflowError: At least one job did not complete successfully.
Hi @kangjiajinlong
Hmm.. that's still not a nice stack trace (i.e., not easy to tell from this what is going wrong).
Could you try running the command manually to see wether you get any error
scenicplus grn_inference TF_to_gene \
--multiome_mudata_fname /home/jkang5/sc163/scenicplus/outs/ACC_GEX.h5mu \
--tf_names /home/jkang5/sc163/scenicplus/outs/tf_names.txt \
--temp_dir /home/jkang5/sc163/scenicplus/temp \
--out_tf_to_gene_adjacencies /home/jkang5/sc163/scenicplus/outs/tf_to_gene_adj.tsv \
--method GBM \
--n_cpu 4 \
--seed 666
Best,
Seppe