Mike Lin comments

Results 150 comments of


Mike Lin

deal with duplicate keys in runtime/meta sections

https://github.com/DataBiosphere/topmed-workflows/issues/92 https://github.com/gatk-workflows/five-dollar-genome-analysis-pipeline/issues/18

Keep going with independent tasks

There's a [config option](https://github.com/chanzuckerberg/miniwdl/blob/41af21a9b3aeca3b4eeedf5bf4d8e586570c4c35/WDL/runtime/config_templates/default.cfg#L25-L28) `[scheduler] fail_fast = false` / env `MINIWDL__SCHEDULER__FAIL_FAST=false` that should do this. I don't think we have a test case for it with AWS specifically, but that...

Keep going with independent tasks

That [default.cfg](https://github.com/chanzuckerberg/miniwdl/blob/main/WDL/runtime/config_templates/default.cfg) is commented extensively, but the [docs on configuration](https://github.com/chanzuckerberg/miniwdl/blob/main/docs/runner_reference.md#configuration) do a really mediocre job of linking out to it -- that's probably the low-hanging fruit

Keep going with independent tasks

Another likely problem is that AGC doesn't yet make it super convenient to set the more-advanced config options (that don't have dedicated command-line arguments). https://github.com/aws/amazon-genomics-cli/pull/420 would help with that dankly....

let task output files outside its working directory

Example to noodle on -- a straightforward `samtools index` or `tabix` task. The input files are mounted a bit outside of the working directory. When we run the indexing program,...

let task output files outside its working directory

Possible heuristic for index files: following the current directory assortment pass, if (i) a file Idx has a basename which is equal to the basename of another file Dat plus...

docker tasks occasionally hang when worker goes down

Some issues specific to worker node shutdown: https://github.com/moby/moby/issues/34280 https://github.com/moby/moby/issues/34122 Also notable: docker moves a task to the "orphaned" state only after a node has been down for 24h (!): https://github.com/docker/swarmkit/blob/ebe39a32e3ed4c3a3783a02c11cccf388818694c/manager/dispatcher/dispatcher.go#L50-L53