unable to run conda as standalone application

[suminwei2772]

Hi, I'm trying to run dropSeqPipe, and have followed the steps to install miniconda3. When I try to run the following error:

snakemake --use-conda --cores 4 --directory /projects/lwei_prj/Dolomite

Building DAG of jobs... CreateCondaEnvironmentException: Unable to check conda version: ERROR: The install method you used for conda--probably either pip install conda or easy_install conda--is not compatible with using conda as an application. If your intention is to install conda as a standalone application, currently supported install methods include the Anaconda installer and the miniconda installer. You can download the miniconda installer from https://conda.io/miniconda.html.

File "/home/lwei/miniconda3/lib/python3.7/site-packages/snakemake/deployment/conda.py", line 232, in create File "/home/lwei/miniconda3/lib/python3.7/site-packages/snakemake/deployment/conda.py", line 343, in new File "/home/lwei/miniconda3/lib/python3.7/site-packages/snakemake/deployment/conda.py", line 356, in init File "/home/lwei/miniconda3/lib/python3.7/site-packages/snakemake/deployment/conda.py", line 411, in _check

Would I need to install conda another way?

[seb-mueller]

It look like the installed conda is not in your PATH and you might have another version of conda which is in the PATH. Could you try echo $PATH? Does it include the conda installation directory? Also, could you try follow the suggestions here for troubleshooting?

[suminwei2772]

Did echo $PATH, and looks like it does include the conda installation directory (/projects/lwei_prj/Dolomite/miniconda3/bin):

/projects/lwei_prj/python_virtual_env/bin:/projects/lwei_prj/Dolomite/miniconda3/bin:/projects/lwei_prj/Dolomite/miniconda3/condabin:.:/projects/marralab_prj/lwei/AAML1031/scripts/samtools-1.9:.:/projects/lwei_prj/python_virtual_env_arboreto/bin/python:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/projects/lwei_prj/Dolomite/miniconda3/bin

[seb-mueller]

Indeed, the path seems correct. Strange. I still think there is something wrong with conda in general.Could you try to install something in conda, e.g. conda install samtools or so? Failing this, could you have a look what conda info gives you.

[suminwei2772]

Hi @seb-mueller, So with some help from Richard Corbett (also posted an issue) I managed to get conda working by adding the miniconda3 install (/projects/lwei_prj/Dolomite/miniconda3/bin) to the beginning of my PATH, and doing the following:

conda create -n dropSeqPipe conda install -c bioconda -c conda-forge snakemake conda activate dropSeqPipe

I re-started the run, and I'm now getting the following error when running snakemake --use-conda --cores 20 --directory /projects/lwei_prj/Dolomite/dropSeqPipe2

Building DAG of jobs... Using shell: /usr/bin/bash Provided cores: 20 Rules claiming more threads will be scaled down. Job counts: count jobs 4 DetectBeadSubstitutionErrors 3 MergeBamAlignment 3 STAR_align 4 SingleCellRnaSeqMetricsCollector 4 TagReadWithGeneExon 1 all 4 bam_hist 4 bead_errors_metrics 8 convert_long_to_mtx 4 extract_reads_expression 4 extract_umi_expression 2 merge_long 1 multiqc_star 3 pigz_unmapped 1 plot_adapter_content 4 plot_knee_plot 4 plot_rna_metrics 1 plot_yield 4 repair_barcodes 1 summary_stats 1 violine_plots 65

[Tue Feb 11 23:01:58 2020] rule repair_barcodes: input: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/Aligned.merged.bam, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/barcode_ref.pkl, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/barcode_ext_ref.pkl, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/empty_barcode_mapping.pkl output: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/Aligned.repaired.bam, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/barcode_mapping_counts.pkl jobid: 108 wildcards: results_dir=/projects/lwei_prj/Dolomite/dropSeqPipe2/results, sample=sample4

[Tue Feb 11 23:01:58 2020] localrule plot_adapter_content: input: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/logs/cutadapt/sample1.clean_qc.csv, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/logs/cutadapt/sample2.clean_qc.csv, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/logs/cutadapt/sample3.clean_qc.csv, /projects/lwei_prj/Dolomite/dropSeqPipe2/results/logs/cutadapt/sample4.clean_qc.csv output: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/plots/adapter_content.pdf jobid: 4 wildcards: results_dir=/projects/lwei_prj/Dolomite/dropSeqPipe2/results

Activating conda environment: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/c7fcf3fb Activating conda environment: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

filter, lag

The following objects are masked from ‘package:base’:

intersect, setdiff, setequal, union

Loading required package: viridisLite Error in mutate_impl(.data, dots) : Evaluation error: as_dictionary() is defunct as of rlang 0.3.0. Please use as_data_pronoun() instead. Calls: %>% ... <Anonymous> -> mutate -> mutate.tbl_df -> mutate_impl -> .Call Execution halted [Tue Feb 11 23:02:14 2020] Error in rule plot_adapter_content: jobid: 4 output: /projects/lwei_prj/Dolomite/dropSeqPipe2/results/plots/adapter_content.pdf conda-env: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b

RuleException: CalledProcessError in line 120 of /projects/lwei_prj/Dolomite/dropSeqPipe2/rules/filter.smk: Command 'source /projects/lwei_prj/Dolomite/miniconda3/bin/activate '/projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b'; set -euo pipefail; Rscript --vanilla /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/scripts/tmpi_2c5y5g.plot_adapter_content.R' returned non-zero exit status 1. File "/projects/lwei_prj/Dolomite/dropSeqPipe2/rules/filter.smk", line 120, in __rule_plot_adapter_content File "/projects/lwei_prj/Dolomite/miniconda3/envs/dropSeqPipe/lib/python3.6/concurrent/futures/thread.py", line 56, in run Removing temporary output file /projects/lwei_prj/Dolomite/dropSeqPipe2/results/samples/sample4/Aligned.merged.bam. [Tue Feb 11 23:12:13 2020] Finished job 108. 1 of 65 steps (2%) done Shutting down, this might take some time. Exiting because a job execution failed. Look above for error message Complete log: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/log/2020-02-11T230153.534247.snakemake.log

Do you know what might be wrong? Thank you, Lisa

[seb-mueller]

First of all, conda seem to work now, and this seems to be another error.

This is a bit unexpected and probably has to do how conda resolves package dependencies (which we don't have full control over). I suspect the dplyr and rlang versions don't like each other, could you try activating the environment in question:

conda active /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b
# start R (which is part of the environment)
R

within R, run the following and report back the results:

library(ggplot2)
library(dplyr)
library(viridis)
sessionInfo()

Also, which version of dropseqpipe do you use?

[suminwei2772]

I typed in conda activate /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b (/projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b) [lwei@marrahost02 dropSeqPipe2]$ R I then get the following:

R version 3.4.1 (2017-06-30) -- "Single Candle"
Copyright (C) 2017 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(ggplot2)

> library(dplyr)

Attaching package: ‘dplyr’

The following objects are masked from ‘package:stats’:

    filter, lag

The following objects are masked from ‘package:base’:

    intersect, setdiff, setequal, union

> library(viridis)
Loading required package: viridisLite
> sessionInfo()
R version 3.4.1 (2017-06-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: CentOS release 6.7 (Final)

Matrix products: default
BLAS: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b/lib/R/lib/libRblas.so
LAPACK: /projects/lwei_prj/Dolomite/dropSeqPipe2/.snakemake/conda/d3403e9b/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_CA.UTF-8        LC_COLLATE=en_CA.UTF-8    
 [5] LC_MONETARY=en_CA.UTF-8    LC_MESSAGES=en_CA.UTF-8   
 [7] LC_PAPER=en_CA.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_CA.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] viridis_0.5.1     viridisLite_0.3.0 dplyr_0.7.4       ggplot2_3.1.0    

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.15     withr_2.1.2      crayon_1.3.4     assertthat_0.2.0
 [5] grid_3.4.1       plyr_1.8.4       R6_2.2.2         gtable_0.2.0    
 [9] magrittr_1.5     scales_1.0.0     pillar_1.3.0     rlang_0.3.0.1   
[13] lazyeval_0.2.1   bindrcpp_0.2     glue_1.3.0       munsell_0.5.0   
[17] compiler_3.4.1   pkgconfig_2.0.2  colorspace_1.3-2 bindr_0.1.1     
[21] gridExtra_2.3    tibble_1.4.2

[suminwei2772]

Also, I cloned the dropSeqPipe from Github using git clone https://github.com/Hoohm/dropSeqPipe.git I'm not sure how I may be able to find out which version this is?

[Hoohm]

Hello @suminwei2772 would you be ok to try the singularity approach instead of the conda one?

[seb-mueller]

I'm pretty sure you are on version 0.4, i.e. the current master. It is concerning that this is not working for you. Strangely, your dplyr version is 0.7.4 even though it should be 0.7.6. Not sure why this is, maybe it is interfering with a R local install, you could check by typing .libPaths() in the session above. Otherwise, I would also suggest, failing @Hoohm suggestion, to try out the develop version, i.e. git checkout develop and rerun the pipeline.