toil icon indicating copy to clipboard operation
toil copied to clipboard

Published 20 hours ago • verified publisher icon DataBiosphere

--debugWorker flag causes the job to restart infinitely

Open multimeric opened this issue 6 years ago • 4 comments

If I run a failing job without --debugWorker, it fails as normal:

michael-Latitude-7480 2019-07-15 14:33:03,063 MainThread INFO toil.leader: Job ended: 'BamToFastq' kind-BamToFastq/instancetxvhy0p2
michael-Latitude-7480 2019-07-15 14:33:03,064 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'BamToFastq' kind-BamToFastq/instancetxvhy0p2
michael-Latitude-7480 2019-07-15 14:33:03,064 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
michael-Latitude-7480 2019-07-15 14:33:03,064 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2    INFO:toil:Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:33:03,064 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2    WARNING:toil.resource:'JTRES_687159ac69c1311b2fa14309e95be8a6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
michael-Latitude-7480 2019-07-15 14:33:03,064 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2    ERROR:toil.lib.docker:Docker had non-zero exit.  Check your command: 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/alignment.bam'
michael-Latitude-7480 2019-07-15 14:33:03,064 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2    Traceback (most recent call last):
michael-Latitude-7480 2019-07-15 14:33:03,064 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2      File "/home/michael/Programming/toil/src/toil/worker.py", line 337, in workerScript
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2      File "/home/michael/Programming/toil/src/toil/job.py", line 1389, in _runner
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2        returnValues = self._run(jobGraph, fileStore)
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2      File "/home/michael/Programming/toil/src/toil/job.py", line 1328, in _run
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2        return self.run(fileStore)
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2      File "/home/michael/Programming/1000gMultiQc/workflow.py", line 73, in run
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2        parameters=parameters
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2      File "/home/michael/Programming/toil/src/toil/lib/docker.py", line 232, in apiDockerCall
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2        **kwargs)
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2      File "/home/michael/Programming/1000gMultiQc/venv/lib/python3.6/site-packages/docker/models/containers.py", line 734, in run
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2        raise ContainerError(container, exit_status, command, image, out)
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2    docker.errors.ContainerError: Command 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/alignment.bam' in image 'quay.io/biocontainers/samtools:1.8--4' returned non-zero exit status 1: b'[E::hts_open_format] Failed to open file /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/alignment.bam\nsamtools bam2fq: Cannot read file "/tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcg7nuqq_/573d1468-e853-44dc-b49a-0147c7ade713/treooy91n/alignment.bam": No such file or directory\n'
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2    ERROR:toil.worker:Exiting the worker because of a failed job on host michael-Latitude-7480
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: kind-BamToFastq/instancetxvhy0p2    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'BamToFastq' kind-BamToFastq/instancetxvhy0p2 with ID kind-BamToFastq/instancetxvhy0p2 to 0
michael-Latitude-7480 2019-07-15 14:33:03,065 MainThread WARNING toil.leader: Job 'BamToFastq' kind-BamToFastq/instancetxvhy0p2 with ID kind-BamToFastq/instancetxvhy0p2 is completely failed
michael-Latitude-7480 2019-07-15 14:33:03,210 MainThread INFO toil.leader: Job ended: 'BamToFastq' kind-BamToFastq/instance5cekp_wm
michael-Latitude-7480 2019-07-15 14:33:03,210 MainThread WARNING toil.leader: The job seems to have left a log file, indicating failure: 'BamToFastq' kind-BamToFastq/instance5cekp_wm
michael-Latitude-7480 2019-07-15 14:33:03,210 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm    INFO:toil.worker:---TOIL WORKER OUTPUT LOG---
michael-Latitude-7480 2019-07-15 14:33:03,210 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm    INFO:toil:Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:33:03,210 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm    WARNING:toil.resource:'JTRES_687159ac69c1311b2fa14309e95be8a6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm    ERROR:toil.lib.docker:Docker had non-zero exit.  Check your command: 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/alignment.bam'
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm    Traceback (most recent call last):
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm      File "/home/michael/Programming/toil/src/toil/worker.py", line 337, in workerScript
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm        job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm      File "/home/michael/Programming/toil/src/toil/job.py", line 1389, in _runner
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm        returnValues = self._run(jobGraph, fileStore)
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm      File "/home/michael/Programming/toil/src/toil/job.py", line 1328, in _run
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm        return self.run(fileStore)
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm      File "/home/michael/Programming/1000gMultiQc/workflow.py", line 73, in run
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm        parameters=parameters
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm      File "/home/michael/Programming/toil/src/toil/lib/docker.py", line 232, in apiDockerCall
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm        **kwargs)
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm      File "/home/michael/Programming/1000gMultiQc/venv/lib/python3.6/site-packages/docker/models/containers.py", line 734, in run
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm        raise ContainerError(container, exit_status, command, image, out)
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm    docker.errors.ContainerError: Command 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/alignment.bam' in image 'quay.io/biocontainers/samtools:1.8--4' returned non-zero exit status 1: b'[E::hts_open_format] Failed to open file /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/alignment.bam\nsamtools bam2fq: Cannot read file "/tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp8h5soiyv/910524e0-7cea-4a29-813d-c1c6ff9739ad/ttktntr_c/alignment.bam": No such file or directory\n'
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm    ERROR:toil.worker:Exiting the worker because of a failed job on host michael-Latitude-7480
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: kind-BamToFastq/instance5cekp_wm    WARNING:toil.jobGraph:Due to failure we are reducing the remaining retry count of job 'BamToFastq' kind-BamToFastq/instance5cekp_wm with ID kind-BamToFastq/instance5cekp_wm to 0
michael-Latitude-7480 2019-07-15 14:33:03,211 MainThread WARNING toil.leader: Job 'BamToFastq' kind-BamToFastq/instance5cekp_wm with ID kind-BamToFastq/instance5cekp_wm is completely failed
michael-Latitude-7480 2019-07-15 14:33:11,215 MainThread INFO toil.leader: Finished toil run with 5 failed jobs.
michael-Latitude-7480 2019-07-15 14:33:11,215 MainThread INFO toil.leader: Failed jobs at end of the run: 'DownloadS3File' kind-DownloadS3File/instancel7arb9gg 'DownloadS3File' kind-DownloadS3File/instance33oao9cj 'BamToFastq' kind-BamToFastq/instancetxvhy0p2 'BamToFastq' kind-BamToFastq/instance5cekp_wm 'ProcessSample' kind-ListSamples/instanceb56jbtdw
Traceback (most recent call last):
  File "/home/michael/Programming/1000gMultiQc/workflow.py", line 104, in <module>
    toil.restart()
  File "/home/michael/Programming/toil/src/toil/common.py", line 813, in restart
    return self._runMainLoop(rootJobGraph)
  File "/home/michael/Programming/toil/src/toil/common.py", line 1054, in _runMainLoop
    jobCache=self._jobCache).run()
  File "/home/michael/Programming/toil/src/toil/leader.py", line 246, in run
    raise FailedJobsException(self.config.jobStore, self.toilState.totalFailedJobs, self.jobStore)
toil.leader.FailedJobsException

Process finished with exit code 1

However, with the --debugWorker flag, it repeats infinitely. In this case I stopped it after about 5 loops, but it will keep going forever otherwise:

michael-Latitude-7480 2019-07-15 14:34:00,353 MainThread DEBUG past.translation: Running find_module: avro...
michael-Latitude-7480 2019-07-15 14:34:00,353 MainThread DEBUG past.translation: Running load_module for avro...
michael-Latitude-7480 2019-07-15 14:34:00,353 MainThread DEBUG past.translation: Autoconverting avro ...
michael-Latitude-7480 2019-07-15 14:34:00,353 MainThread DEBUG past.translation: Running find_module: avro.schema...
michael-Latitude-7480 2019-07-15 14:34:00,353 MainThread DEBUG past.translation: Running load_module for avro.schema...
michael-Latitude-7480 2019-07-15 14:34:00,353 MainThread DEBUG past.translation: Autoconverting avro.schema ...
michael-Latitude-7480 2019-07-15 14:34:00,606 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxCores to CPU count of system (4).
michael-Latitude-7480 2019-07-15 14:34:00,606 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxMemory to physically available memory (25154138112).
michael-Latitude-7480 2019-07-15 14:34:00,606 MainThread WARNING toil.batchSystems.singleMachine: Limiting maxDisk to physically available disk (110291382272).
michael-Latitude-7480 2019-07-15 14:34:00,607 MainThread WARNING toil.jobStores.fileJobStore: Job Dir: instancel7arb9gg
michael-Latitude-7480 2019-07-15 14:34:00,607 MainThread WARNING toil.jobStores.fileJobStore: Job Dir: instance33oao9cj
michael-Latitude-7480 2019-07-15 14:34:00,607 MainThread WARNING toil.jobStores.fileJobStore: Job Dir: instancetxvhy0p2
michael-Latitude-7480 2019-07-15 14:34:00,607 MainThread WARNING toil.jobStores.fileJobStore: Job Dir: instance5cekp_wm
michael-Latitude-7480 2019-07-15 14:34:00,607 MainThread WARNING toil.jobStores.fileJobStore: Job Dir: instanceb56jbtdw
michael-Latitude-7480 2019-07-15 14:34:00,607 MainThread CRITICAL toil.jobStores.abstractJobStore: Repairing job: kind-BamToFastq/instance5cekp_wm
michael-Latitude-7480 2019-07-15 14:34:00,608 MainThread CRITICAL toil.jobStores.abstractJobStore: Repairing job: kind-BamToFastq/instancetxvhy0p2
michael-Latitude-7480 2019-07-15 14:34:00,608 MainThread INFO toil: Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:34:00,612 MainThread INFO toil.worker: ---TOIL WORKER OUTPUT LOG---
michael-Latitude-7480 2019-07-15 14:34:00,612 MainThread INFO toil: Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:34:00,612 MainThread WARNING toil.resource: The localize() method should only be invoked on a worker.
michael-Latitude-7480 2019-07-15 14:34:00,612 MainThread WARNING toil.resource: 'JTRES_687159ac69c1311b2fa14309e95be8a6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
michael-Latitude-7480 2019-07-15 14:34:01,600 MainThread ERROR toil.lib.docker: Docker had non-zero exit.  Check your command: 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/alignment.bam'
Traceback (most recent call last):
  File "/home/michael/Programming/toil/src/toil/worker.py", line 337, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1389, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1328, in _run
    return self.run(fileStore)
  File "/home/michael/Programming/1000gMultiQc/workflow.py", line 73, in run
    parameters=parameters
  File "/home/michael/Programming/toil/src/toil/lib/docker.py", line 232, in apiDockerCall
    **kwargs)
  File "/home/michael/Programming/1000gMultiQc/venv/lib/python3.6/site-packages/docker/models/containers.py", line 734, in run
    raise ContainerError(container, exit_status, command, image, out)
docker.errors.ContainerError: Command 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/alignment.bam' in image 'quay.io/biocontainers/samtools:1.8--4' returned non-zero exit status 1: b'[E::hts_open_format] Failed to open file /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/alignment.bam\nsamtools bam2fq: Cannot read file "/tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg611tl5c/39054f50-c259-48dd-8205-141f686b2bba/tu44qvxey/alignment.bam": No such file or directory\n'
michael-Latitude-7480 2019-07-15 14:34:01,604 MainThread ERROR toil.worker: Exiting the worker because of a failed job on host michael-Latitude-7480
michael-Latitude-7480 2019-07-15 14:34:01,604 MainThread WARNING toil.jobGraph: Due to failure we are reducing the remaining retry count of job 'BamToFastq' kind-BamToFastq/instance5cekp_wm with ID kind-BamToFastq/instance5cekp_wm to 0
michael-Latitude-7480 2019-07-15 14:34:01,605 MainThread INFO toil.leader: Issued job 'BamToFastq' kind-BamToFastq/instance5cekp_wm with job batch system ID: 0 and cores: 1, disk: 5.0 G, and memory: 2.0 G
michael-Latitude-7480 2019-07-15 14:34:01,606 MainThread INFO toil.worker: ---TOIL WORKER OUTPUT LOG---
michael-Latitude-7480 2019-07-15 14:34:01,606 MainThread INFO toil: Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:34:01,606 MainThread WARNING toil.resource: The localize() method should only be invoked on a worker.
michael-Latitude-7480 2019-07-15 14:34:01,606 MainThread WARNING toil.resource: 'JTRES_687159ac69c1311b2fa14309e95be8a6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
michael-Latitude-7480 2019-07-15 14:34:02,581 MainThread ERROR toil.lib.docker: Docker had non-zero exit.  Check your command: 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/alignment.bam'
Traceback (most recent call last):
  File "/home/michael/Programming/toil/src/toil/worker.py", line 337, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1389, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1328, in _run
    return self.run(fileStore)
  File "/home/michael/Programming/1000gMultiQc/workflow.py", line 73, in run
    parameters=parameters
  File "/home/michael/Programming/toil/src/toil/lib/docker.py", line 232, in apiDockerCall
    **kwargs)
  File "/home/michael/Programming/1000gMultiQc/venv/lib/python3.6/site-packages/docker/models/containers.py", line 734, in run
    raise ContainerError(container, exit_status, command, image, out)
docker.errors.ContainerError: Command 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/alignment.bam' in image 'quay.io/biocontainers/samtools:1.8--4' returned non-zero exit status 1: b'[E::hts_open_format] Failed to open file /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/alignment.bam\nsamtools bam2fq: Cannot read file "/tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpga04ti9e/e529f5fd-805e-4f35-8cde-3d36d3a3de15/tq2wikq6n/alignment.bam": No such file or directory\n'
michael-Latitude-7480 2019-07-15 14:34:02,589 MainThread ERROR toil.worker: Exiting the worker because of a failed job on host michael-Latitude-7480
michael-Latitude-7480 2019-07-15 14:34:02,590 MainThread WARNING toil.jobGraph: Due to failure we are reducing the remaining retry count of job 'BamToFastq' kind-BamToFastq/instancetxvhy0p2 with ID kind-BamToFastq/instancetxvhy0p2 to 0
michael-Latitude-7480 2019-07-15 14:34:02,591 MainThread INFO toil.leader: Issued job 'BamToFastq' kind-BamToFastq/instancetxvhy0p2 with job batch system ID: 1 and cores: 1, disk: 5.0 G, and memory: 2.0 G
michael-Latitude-7480 2019-07-15 14:34:02,591 MainThread INFO toil.leader: Job ended: 'BamToFastq' kind-BamToFastq/instance5cekp_wm
michael-Latitude-7480 2019-07-15 14:34:02,594 MainThread INFO toil.worker: ---TOIL WORKER OUTPUT LOG---
michael-Latitude-7480 2019-07-15 14:34:02,594 MainThread INFO toil: Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:34:02,594 MainThread WARNING toil.resource: The localize() method should only be invoked on a worker.
michael-Latitude-7480 2019-07-15 14:34:02,595 MainThread WARNING toil.resource: 'JTRES_687159ac69c1311b2fa14309e95be8a6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
michael-Latitude-7480 2019-07-15 14:34:03,554 MainThread ERROR toil.lib.docker: Docker had non-zero exit.  Check your command: 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/alignment.bam'
Traceback (most recent call last):
  File "/home/michael/Programming/toil/src/toil/worker.py", line 337, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1389, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1328, in _run
    return self.run(fileStore)
  File "/home/michael/Programming/1000gMultiQc/workflow.py", line 73, in run
    parameters=parameters
  File "/home/michael/Programming/toil/src/toil/lib/docker.py", line 232, in apiDockerCall
    **kwargs)
  File "/home/michael/Programming/1000gMultiQc/venv/lib/python3.6/site-packages/docker/models/containers.py", line 734, in run
    raise ContainerError(container, exit_status, command, image, out)
docker.errors.ContainerError: Command 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/alignment.bam' in image 'quay.io/biocontainers/samtools:1.8--4' returned non-zero exit status 1: b'[E::hts_open_format] Failed to open file /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/alignment.bam\nsamtools bam2fq: Cannot read file "/tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpg4vd4lrb/fead62f1-65cf-4d92-9fd9-300786898c5e/t056k4ko3/alignment.bam": No such file or directory\n'
michael-Latitude-7480 2019-07-15 14:34:03,566 MainThread ERROR toil.worker: Exiting the worker because of a failed job on host michael-Latitude-7480
michael-Latitude-7480 2019-07-15 14:34:03,569 MainThread WARNING toil.jobGraph: Due to failure we are reducing the remaining retry count of job 'BamToFastq' kind-BamToFastq/instance5cekp_wm with ID kind-BamToFastq/instance5cekp_wm to 0
michael-Latitude-7480 2019-07-15 14:34:03,570 MainThread INFO toil.leader: Issued job 'BamToFastq' kind-BamToFastq/instance5cekp_wm with job batch system ID: 2 and cores: 1, disk: 5.0 G, and memory: 2.0 G
michael-Latitude-7480 2019-07-15 14:34:03,571 MainThread INFO toil.leader: Job ended: 'BamToFastq' kind-BamToFastq/instancetxvhy0p2
michael-Latitude-7480 2019-07-15 14:34:03,574 MainThread INFO toil.worker: ---TOIL WORKER OUTPUT LOG---
michael-Latitude-7480 2019-07-15 14:34:03,574 MainThread INFO toil: Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:34:03,575 MainThread WARNING toil.resource: The localize() method should only be invoked on a worker.
michael-Latitude-7480 2019-07-15 14:34:03,575 MainThread WARNING toil.resource: 'JTRES_687159ac69c1311b2fa14309e95be8a6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
michael-Latitude-7480 2019-07-15 14:34:04,615 MainThread ERROR toil.lib.docker: Docker had non-zero exit.  Check your command: 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/alignment.bam'
Traceback (most recent call last):
  File "/home/michael/Programming/toil/src/toil/worker.py", line 337, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1389, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1328, in _run
    return self.run(fileStore)
  File "/home/michael/Programming/1000gMultiQc/workflow.py", line 73, in run
    parameters=parameters
  File "/home/michael/Programming/toil/src/toil/lib/docker.py", line 232, in apiDockerCall
    **kwargs)
  File "/home/michael/Programming/1000gMultiQc/venv/lib/python3.6/site-packages/docker/models/containers.py", line 734, in run
    raise ContainerError(container, exit_status, command, image, out)
docker.errors.ContainerError: Command 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/alignment.bam' in image 'quay.io/biocontainers/samtools:1.8--4' returned non-zero exit status 1: b'[E::hts_open_format] Failed to open file /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/alignment.bam\nsamtools bam2fq: Cannot read file "/tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmp5vsdhgzy/584b91de-5222-45d7-abd8-e2e882cfe49d/tinz541kv/alignment.bam": No such file or directory\n'
michael-Latitude-7480 2019-07-15 14:34:04,619 MainThread ERROR toil.worker: Exiting the worker because of a failed job on host michael-Latitude-7480
michael-Latitude-7480 2019-07-15 14:34:04,619 MainThread WARNING toil.jobGraph: Due to failure we are reducing the remaining retry count of job 'BamToFastq' kind-BamToFastq/instancetxvhy0p2 with ID kind-BamToFastq/instancetxvhy0p2 to 0
michael-Latitude-7480 2019-07-15 14:34:04,620 MainThread INFO toil.leader: Issued job 'BamToFastq' kind-BamToFastq/instancetxvhy0p2 with job batch system ID: 3 and cores: 1, disk: 5.0 G, and memory: 2.0 G
michael-Latitude-7480 2019-07-15 14:34:04,620 MainThread INFO toil.leader: Job ended: 'BamToFastq' kind-BamToFastq/instance5cekp_wm
michael-Latitude-7480 2019-07-15 14:34:04,620 MainThread INFO toil.worker: ---TOIL WORKER OUTPUT LOG---
michael-Latitude-7480 2019-07-15 14:34:04,621 MainThread INFO toil: Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:34:04,621 MainThread WARNING toil.resource: The localize() method should only be invoked on a worker.
michael-Latitude-7480 2019-07-15 14:34:04,621 MainThread WARNING toil.resource: 'JTRES_687159ac69c1311b2fa14309e95be8a6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
michael-Latitude-7480 2019-07-15 14:34:05,667 MainThread ERROR toil.lib.docker: Docker had non-zero exit.  Check your command: 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/alignment.bam'
Traceback (most recent call last):
  File "/home/michael/Programming/toil/src/toil/worker.py", line 337, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1389, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1328, in _run
    return self.run(fileStore)
  File "/home/michael/Programming/1000gMultiQc/workflow.py", line 73, in run
    parameters=parameters
  File "/home/michael/Programming/toil/src/toil/lib/docker.py", line 232, in apiDockerCall
    **kwargs)
  File "/home/michael/Programming/1000gMultiQc/venv/lib/python3.6/site-packages/docker/models/containers.py", line 734, in run
    raise ContainerError(container, exit_status, command, image, out)
docker.errors.ContainerError: Command 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/alignment.bam' in image 'quay.io/biocontainers/samtools:1.8--4' returned non-zero exit status 1: b'[E::hts_open_format] Failed to open file /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/alignment.bam\nsamtools bam2fq: Cannot read file "/tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpcw2tb4b7/30286c3b-858f-43af-92f3-39cb844e7c28/t7tt2j80t/alignment.bam": No such file or directory\n'
michael-Latitude-7480 2019-07-15 14:34:05,671 MainThread ERROR toil.worker: Exiting the worker because of a failed job on host michael-Latitude-7480
michael-Latitude-7480 2019-07-15 14:34:05,671 MainThread WARNING toil.jobGraph: Due to failure we are reducing the remaining retry count of job 'BamToFastq' kind-BamToFastq/instance5cekp_wm with ID kind-BamToFastq/instance5cekp_wm to 0
michael-Latitude-7480 2019-07-15 14:34:05,672 MainThread INFO toil.leader: Issued job 'BamToFastq' kind-BamToFastq/instance5cekp_wm with job batch system ID: 4 and cores: 1, disk: 5.0 G, and memory: 2.0 G
michael-Latitude-7480 2019-07-15 14:34:05,672 MainThread INFO toil.leader: Job ended: 'BamToFastq' kind-BamToFastq/instancetxvhy0p2
michael-Latitude-7480 2019-07-15 14:34:05,672 MainThread INFO toil.worker: ---TOIL WORKER OUTPUT LOG---
michael-Latitude-7480 2019-07-15 14:34:05,672 MainThread INFO toil: Running Toil version 3.21.0a1-042a5529449e4e4d8776fb6da8edcf0ac90c7132-dirty.
michael-Latitude-7480 2019-07-15 14:34:05,673 MainThread WARNING toil.resource: The localize() method should only be invoked on a worker.
michael-Latitude-7480 2019-07-15 14:34:05,673 MainThread WARNING toil.resource: 'JTRES_687159ac69c1311b2fa14309e95be8a6' may exist, but is not yet referenced by the worker (KeyError from os.environ[]).
michael-Latitude-7480 2019-07-15 14:34:06,729 MainThread ERROR toil.lib.docker: Docker had non-zero exit.  Check your command: 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/alignment.bam'
Traceback (most recent call last):
  File "/home/michael/Programming/toil/src/toil/worker.py", line 337, in workerScript
    job._runner(jobGraph=jobGraph, jobStore=jobStore, fileStore=fileStore, defer=defer)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1389, in _runner
    returnValues = self._run(jobGraph, fileStore)
  File "/home/michael/Programming/toil/src/toil/job.py", line 1328, in _run
    return self.run(fileStore)
  File "/home/michael/Programming/1000gMultiQc/workflow.py", line 73, in run
    parameters=parameters
  File "/home/michael/Programming/toil/src/toil/lib/docker.py", line 232, in apiDockerCall
    **kwargs)
  File "/home/michael/Programming/1000gMultiQc/venv/lib/python3.6/site-packages/docker/models/containers.py", line 734, in run
    raise ContainerError(container, exit_status, command, image, out)
docker.errors.ContainerError: Command 'fastq -0 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/r0.fastq -1 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/r1.fastq -2 /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/r2.fastq /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/alignment.bam' in image 'quay.io/biocontainers/samtools:1.8--4' returned non-zero exit status 1: b'[E::hts_open_format] Failed to open file /tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/alignment.bam\nsamtools bam2fq: Cannot read file "/tmp/toil-a6d839df-88f8-4cd8-94d6-0ed7f88c53c6-25e1ce7eb26c40119f94b4241a8c703b/tmpo7gdhlrd/c09bac4d-520a-4687-969c-ef72a77ccc4c/tebc3kiss/alignment.bam": No such file or directory\n'

┆Issue is synchronized with this Jira Story ┆Epic: Improve debugging experience ┆Issue Number: TOIL-396

multimeric avatar Jul 15 '19 04:07 multimeric

This issue is known to me, yeah. Since I'm only ever debugging something that fails I never got around to fixing it 😂

mr-c avatar Jul 15 '19 14:07 mr-c

Well yes I can see why fixing this wouldn't be high priority, since you can just remove the --debugWorker flag. But this had me perplexed for a while, until I realised the culprit.

If nothing else, having this issue should document the bug if anyone encounters it later on.

multimeric avatar Jul 15 '19 15:07 multimeric

@stxue1 still sees this happening when using --debugWorker under PyCharm.

adamnovak avatar Oct 03 '23 17:10 adamnovak

➤ stxue commented:

I think it is because of this line: https://github.com/DataBiosphere/toil/blob/017f3e1a7441a30ccdd8fc32810322ea8d2f70af/src/toil/batchSystems/singleMachine.py#L488 ( https://github.com/DataBiosphere/toil/blob/017f3e1a7441a30ccdd8fc32810322ea8d2f70af/src/toil/batchSystems/singleMachine.py#L488 )

It looks like the job is always reported as successful in debugWorker mode, so the retryCount never decreases, which is what the innerLoop depends on, causing an infinite loop.

I think ideally, it should retry the same number of times as --retryCount

unito-bot avatar Mar 09 '24 04:03 unito-bot