Nextflow PYSCENIC DSL2 stops running after the first two first two steps

[mt98]

I am running nextflow DSL2 multiruns and first testing it on a small no of runs 2 to see if it is working on a few runs before doing a much larger no of runs. I have a loom file that I created from processed single cell data available online. When I run the nextflow pipeline, it stops after executing the first two steps. I can not understand why the chain does not continue to the next step. I get the Im__run_1__adj.tsv Im__run_2__adj.tsv files but the process does not automatically continue. Any suggestions will be helpful.

executor > slurm (4) [56/b77718] process > scenic:SCENIC:ARBORETO_WITH... [100%] 2 of 2 ✔ [75/58693b] process > scenic:SCENIC:ADD_PEARSON_C... [100%] 2 of 2 ✔ [- ] process > scenic:SCENIC:CISTARGET__MOTIF - [- ] process > scenic:SCENIC:CISTARGET__TRACK - [- ] process > scenic:SCENIC:AUCELL__MOTIF - [- ] process > scenic:SCENIC:AUCELL__TRACK - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MULTI_RUNS_TO... - [- ] process > scenic:SCENIC:MERGE_MOTIF_T... - [- ] process > scenic:SCENIC:VISUALIZE - [- ] process > scenic:SCENIC:PUBLISH_LOOM - [- ] process > scenic:PUBLISH_SCENIC:COMPR... - [- ] process > scenic:PUBLISH_SCENIC:SC__P... - WARN: To render the execution DAG in the required format it is required to install Graphviz -- See

My nextflow.config file is as follows:

manifest { name = 'vib-singlecell-nf/vsn-pipelines' description = 'A repository of pipelines for single-cell data in Nextflow DSL2' homePage = 'https://github.com/vib-singlecell-nf/vsn-pipelines' version = '0.25.0' mainScript = 'main.nf' defaultBranch = 'master' nextflowVersion = '!>=20.10.0' }

params { global { project_name = 'Im' outdir = 'results' species = 'human' genome { assembly = 'hg19' } } misc { test { enabled = false } } utils { container = 'vibsinglecellnf/utils:0.4.0' publish { compressionLevel = 6 annotateWithBatchVariableName = false mode = 'link' } } sc { file_converter { off = 'h5ad' tagCellWithSampleId = true remove10xGEMWell = false useFilteredMatrix = true makeVarIndexUnique = false iff = 'csv' } scenic { container = 'aertslab/pyscenic:0.10.4' scenicoutdir = 'results/scenic' filteredLoom = 'data/Im.loom' scenicOutputLoom = 'SCENIC_output.loom' scenicScopeOutputLoom = 'SCENIC_SCope_output.loom' mode = 'dask_multiprocessing' client_or_address = '' cell_id_attribute = 'CellID' gene_attribute = 'Gene' report_ipynb = '/src/scenic/bin/reports/scenic_report.ipynb' skipReports = false grn { algorithm = 'grnboost2' tfs = '/databases/hs_hgnc_curated_tfs.txt' } cistarget { adj = 'adj.tsv' type = '' rank_threshold = 5000 auc_threshold = 0.05 nes_threshold = 3.0 min_orthologous_identity = 0.0 max_similarity_fdr = 0.001 annotations_fname = '' thresholds = '0.75,0.90' top_n_targets = 50 top_n_regulators = '5,10,50' min_genes = 20 all_modules = false motifsDb = '/databases/hg19__refseq-r45__*feather' motifsAnnotation = '/databases/motifs-v9-nr.hgnc-m0.001-o0.0.tbl' tracksDb = '/databases/encode_20190621__ChIP_seq_transcription_factor.hg19__refseq-r45__*feather' tracksAnnotation = '/databases/encode_project_20190621__ChIP-seq_transcription_factor.homo_sapiens.hg19.bigwig_signal_pvalue.track_to_tf_in_motif_to_tf_format.tsv' } aucell { output = 'aucell_output.loom' rank_threshold = 5000 auc_threshold = 0.05 nes_threshold = 3.0 min_genes_regulon = 5 min_regulon_gene_occurrence = 5 } numRuns = 2 aggregate_features { use_chunking = true output_format = 'csv' compression = 'gzip' } } scope { genome = '' tree { level_1 = '' level_2 = '' level_3 = '' } } } data { loom { file_paths = '/data/Im.loom' suffix = '.loom' } } }

process { executor = 'slurm' queue = 'genomics' clusterOptions = '-A b1042 -t 24:00:00 -e errlog.txt' }

timeline { enabled = true file = 'results/nextflow_reports/execution_timeline.html' }

report { enabled = true file = 'results/nextflow_reports/execution_report.html' }

trace { enabled = true file = 'results/nextflow_reports/execution_trace.txt' }

dag { enabled = true file = 'results/nextflow_reports/pipeline_dag.svg' }

min { enabled = false }

singularity { enabled = true cacheDir = "/projects/work/singularity/" runOptions = '-B /projects:/projects,/projects/data:/data/,/projects/databases:/databases/,/projects/resources:/resources/' }

[Erfan1369]

Hi @mt98

Hope you have find your answer.

I was trying to run pyscenic on CLI using nodes(beluga, graham,..) I'm a little bit naive on creating and using config file to run pyscenic. I have a few simple question about your config file. I would appreciate if you anwered me. 1)shall I use "slurm" as you use here? what is this -A b1042 in cluster option section? 2) Does cacheDir = "/projects/work/singularity/" is the directory you put your own singularity image or it will be pulled automatically by pipeline with using the same cacheDir command here you used? 3) what is this runOption? is it a directory for each files scenic pulled to use?

Thanks,

Erfan

}

[ghuls]

If you don't have a cluster, put the executor to local, else adapt to whatever scheduler your HPC cluster is using. The clusterOptions are additional arguments that you might need to pass to your scheduler to submit jobs (e.g. from which project credits to charge the jobs). See NextFlow documentation: https://www.nextflow.io/docs/latest/executor.html

runOptions are extra options passed to singularity/apptainer, in this case which directories to bind mount in your containers. https://www.nextflow.io/docs/latest/config.html?highlight=runoptions#scope-singularity