docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist

[tetron]

trafficstars

We have a few reports about the container failing to start because one of the bind mounts cannot be found, the strange thing is that these are different invocations on different operating systems.

(env) decagon@Decagons-MacBook-Pro rnaseq-cwl-training-exercises %
cwltool --tmp-outdir-prefix=/Users/decagon/Documents/ifyProjects/outreachy --tmpdir-prefix=/Users/decagon/Documents/ifyProjects/outreachy main.cwl main-input.yaml
INFO /Users/decagon/env/bin/cwltool 3.0.20210319143721
INFO Resolved 'main.cwl' to 'file:///Users/decagon/Documents/ifyProjects/outreachy/rnaseq-cwl-training-exercises/main.cwl'
WARNING Workflow checker warning:
main.cwl:30:11: Source 'alignment' of type ["File", {"type": "array", "items": "File"}] may be
incompatible
main.cwl:36:7: with sink 'bam_sorted' of type "File"
INFO [workflow ] start
INFO [workflow ] starting step STAR
INFO [step STAR] start
INFO [job STAR] /Users/decagon/Documents/ifyProjects/outreachylby4flmu$ docker \
run \
-i \
--mount=type=bind,source=/Users/decagon/Documents/ifyProjects/outreachylby4flmu,target=/muHyRr \
--mount=type=bind,source=/Users/decagon/Documents/ifyProjects/outreachyd4pal9c6,target=/tmp \
--mount=type=bind,source=/Users/decagon/Documents/ifyProjects/outreachy/rnaseq-cwl-training-exercises/rnaseq/raw_fastq/Mov10_oe_1.subset.fq,target=/var/lib/cwl/stg00e6f5c8-8f2b-4f9a-9558-9da84cf9c6d2/Mov10_oe_1.subset.fq,readonly \
--mount=type=bind,source=/Users/decagon/Documents/ifyProjects/outreachy/rnaseq-cwl-training-exercises/hg19-chr1-STAR-index,target=/var/lib/cwl/stg94179c35-f571-448a-b539-6a6ed7f4020e/hg19-chr1-STAR-index,readonly \
--workdir=/muHyRr \
--read-only=true \
--user=501:20 \
--rm \
--env=TMPDIR=/tmp \
--env=HOME=/muHyRr \
--cidfile=/Users/decagon/Documents/ifyProjects/outreachy99r9o6e7/20210407214207-059030.cid \
quay.io/biocontainers/star:2.7.5c--0 \
STAR \
--runMode \
alignReads \
--genomeDir \
/var/lib/cwl/stg94179c35-f571-448a-b539-6a6ed7f4020e/hg19-chr1-STAR-index \
--outSAMunmapped \
Within \
--runThreadN \
4 \
--readFilesIn \
/var/lib/cwl/stg00e6f5c8-8f2b-4f9a-9558-9da84cf9c6d2/Mov10_oe_1.subset.fq \
--outSAMtype \
BAM \
SortedByCoordinate
docker: Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /host_mnt/Users/decagon/Documents/ifyProjects/outreachylby4flmu.

Expected Behavior

Container should run.

Actual Behavior

Container fails with docker error.

Workflow Code

The users who have experienced it see it happening regardless of the workflow being run.

[tetron]

A similar error was reported in #1405

The speculation on that ticket is that there could be a race condition between creating the directory and when it becomes visible to the Docker daemon, due to using a network filesystem.

That could explain the MacOS error as well, as it runs Docker in a VM and uses an internal remote filesystem mount for bind mounts.

However this error was also reported by someone running Ubuntu 20.04, presumably using only local filesystems.

[tetron]

I don't think it is what is happening here, but browsing Docker issues suggests that a mismatch between a case sensitive and insensitive file systems can also cause this problem.

[tetron]

The only idea I have so far is to sniff for this specific error message and automatically retry.

[cure]

Maybe calling os.sync() after the mkdtemp call would help?

[bartns]

For what it's worth, I seem to be reproduce this error on Ubuntu 22.04.

Mounting a folder like this fails: -v /home/bart/test:/test // gives the "Error response from daemon: invalid mount config.... " Error This works: -v /home/bart/test:/home/bart/test

(I'm using docker in docker with a docker.sock mount)