Ruben Vorderman
Ruben Vorderman
Thanks @mlin! The repository is transferred now. I requested codecov access to the organization so we can get coverage reports on Pull requests as well as display a coverage badge.
Huh, I didn't see that, but you are right: ``` 2022-08-08 11:33:53.283 wdl.w:Germline.w:call-JointGenotyping.t:call-gatherGvcfs input :: name: "gvcfFilesIndex", value: ["/mnt/miniwdl_task_container/work/_miniwdl_inputs/13/scatter-0.bed.g.vcf.gz.tbi", "/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/scatter-1.bed.g.vcf.gz.tbi", "/mnt/miniwdl_task_container/work/_miniwdl_inputs/9/scatter-2.bed.g.vcf.gz.tbi", "/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/scatter-0.bed.g.vcf.gz.tbi", "/mnt/miniwdl_task_container/work/_miniwdl_inputs/15/scatter-1.bed.g.vcf.gz.tbi", "/mnt/miniwdl_task_container/work/_miniwdl_inputs/0/scatter-2.bed.g.vcf.gz.tbi"] 2022-08-08 11:33:53.284 wdl.w:Germline.w:call-JointGenotyping.t:call-gatherGvcfs input :: name:...
Solved. So sorry for bothering you with this, turned out to be our own mistake. There was indeed duplication and a simple prefixing of the sample name solved the issue....
I didn't realize I was able to rely on the spec here. Thanks. BioWDL workflows are extensively modularized so we can reuse quite a bit of code. As a result...
Instructions: [deadlock-workflow.tar.gz](https://github.com/chanzuckerberg/miniwdl/files/9349257/deadlock-workflow.tar.gz) ``` wget https://github.com/chanzuckerberg/miniwdl/files/9349257/deadlock-workflow.tar.gz mkdir test-workflow cd test-workflow tar -xvzf ../deadlock-workflow.tar.gz MINIWDL__FILE_IO__ALLOW_ANY_INPUT=true miniwdl run -i tests/integration/Somatic_PairedEnd.json somatic.wdl ``` This will reliably deadlock. It is not backend dependent. I have...
`MINIWDL__SCHEDULER__SUBWORKFLOW_CONCURRENCY=8 MINIWDL__FILE_IO__ALLOW_ANY_INPUT=true miniwdl run -i tests/integration/Somatic_PairedEnd.json somatic.wdl` -> Deadlocks (8 is my core count.) `MINIWDL__SCHEDULER__SUBWORKFLOW_CONCURRENCY=40 MINIWDL__FILE_IO__ALLOW_ANY_INPUT=true miniwdl run -i tests/integration/Somatic_PairedEnd.json somatic.wdl` -> The workflow continues. So your guess was indeed...
I think I got the reason: I have multiple levels of subworkflows. The sampleworklow starts a QC workflow. The reason for this organisation is that a QC workflow is used...
I did some further research and the number of threads python can handle is indeed limited. Using the script mentioned here: https://stackoverflow.com/questions/48970857/maximum-limit-on-number-of-threads-in-python my PC reports a maximum of 30,000 ish...
> I have an idea to initialize a new thread pool for each nesting level, which should still keep the total thread count reasonably under control. That seems to be...