Activate conda environment

[nh13]

I frequently have docker images with one or more conda environments installed. When I run a WDL task, I need to add following to my command each time:

    command <<<
        echo "source activate env" > ~/.bashrc
        source ~/.bashrc
        conda activate bwa;
        set -euxo pipefail
        bwa mem ...

This is more verbose than it needs to be. Is there (a) a way to specify a conda environment can be activated prior to running the command, or (b) have conda available so I can omit the echo and source?

[dpark01]

Do you frequently have docker images with multiple conda environments that you switch between?

I find it much easier to just activate the conda environment in the Dockerfile itself, so that simply docker run-ing the image starts off with everything working in the path and no one even notices that you're even using conda. That seems more generalizable to the docker image's usefulness even outside of WDL.

Example

[nh13]

@dpark01 can you send a link to anexample Docker file? I do frequently have multiple conda environments (see https://github.com/nh13/conda-env-builder) but as long as base is activated I can switch prior to each command.

[dpark01]

Hey sorry it got buried at the end of my last comment:

https://github.com/broadinstitute/viral-core/blob/master/Dockerfile#L31

[nh13]

Your example works when I use docker run -it but not through miniWDL, likely because miniWDL is not using a login shell. Any other thoughts?

[dpark01]

So in addition I think I recall having to manually add the conda environment’s bin to the PATH in the Dockerfile:

https://github.com/broadinstitute/viral-core/blob/master/Dockerfile#L21

I think you’re right that it’s the -it that’s the difference. I think on principle my goal is to create docker images that had the desired executables in the PATH, even in non interactive mode. If you make a docker image that installs samtools via Ubuntu apt get install, you can “docker run myimage samtools” (without -it) and it works, but if you do it with conda, it doesn’t unless you finagle it that way.

Ultimately I don’t think miniwdl itself is the right place to address this. I want my resulting WDL task command block to work the same in Cromwell/Terra and DNAnexus and miniwdl, so fixing it in the docker, if possible, seems the most universal way. Other than that maybe a feature request in OpenWDL, but conda is a little special (and like I said, other package managers don’t require this dance).

[nh13]

@dpark01 I tried adding conda into the path when building my docker image too. It worked just fine with cromwell, so there's something different here between cromwell and miniWDL.

Edit: I have a few ideas, I'll post back if I find something, or close if I don't.

[nh13]

FWIW: snakemake activates conda environments for you ;P

[nh13]

Here's how I fixed it with my local miniWDL install: https://github.com/chanzuckerberg/miniwdl/pull/569

[nh13]

@mlin thanks for the fix in 2c8df3c. Any thoughts on a conda directive?