[AWS] Add runtime attributes

[henriqueribeiro]

Changes:

• Implements RetryStrategy for AWS Batch backend (https://docs.aws.amazon.com/batch/latest/userguide/job_retries.html). Since this is specific for AWS backend a new runtime attribute was created (awsBatchRetryAttempts).
• Add ulimits option (https://docs.aws.amazon.com/batch/latest/userguide/job_definition_parameters.html#containerProperties)

[geertvandeweyer]

I compiled and tested this, and it works correctly. As I'm not familiar with java/scala, I cant provide a full review unfortunately. I did notice some warnings when starting cromwell, but as everything works, maybe that's not a problem ?

2021-03-13 12:17:25,630 WARN - Unrecognized configuration key(s) for AwsBatch: auth, numCreateDefinitionAttempts, default-runtime-attributes.awsBatchRetryAttempts, awsBatchRetryAttempts, filesystems.s3.duplication-strategy, numSubmitAttempts, default-runtime-attributes.scriptBucketName

Thanks by the way ! This was exactly what we were waiting for

[henriqueribeiro]

I just added some documentation.

[Irsan88]

Seems like this is exactly what I need. This branch is not merged yet. Can i clone the repo, checkout this branch and build?

and can you confirm that the right way to set ulimits in the runtime block would e.g. be

runtime { ulimits = [ { "name": string, "softLimit": integer, "hardLimit": integer } ] }

[henriqueribeiro]

@Irsan88 exactly. Checkout this branch and run the wdl with something like this:

"ulimits": [
      {
        "name": "nofile",
        "softLimit": "9000",
        "hardLimit": "9000"
      }
    ],

Make sure that the softLimit and hardLimit are string and not integers

[wdesouza]

I am trying to build this branch but got the error below. Both local (macOS) and cromwell-dev Docker image.

sbt assembly

[error] /code/wom/src/main/scala/wom/views/GraphPrint.scala:113:57: type mismatch;
[error]  found   : java.util.stream.Stream[String]
[error]  required: scala.collection.GenTraversableOnce[?]
[error]           |  ${digraph.links.toList.flatMap(_.dotString.lines).mkString(System.lineSeparator + "  ")}
[error]                                                         ^
[error] /code/wom/src/main/scala/wom/views/GraphPrint.scala:116:57: type mismatch;
[error]  found   : java.util.stream.Stream[String]
[error]  required: scala.collection.GenTraversableOnce[?]
[error]           |  ${digraph.nodes.toList.flatMap(_.dotString.lines).mkString(System.lineSeparator + "  ")}
[error]                                                         ^
[error] /code/wom/src/main/scala/wom/views/GraphPrint.scala:159:49: type mismatch;
[error]  found   : java.util.stream.Stream[String]
[error]  required: scala.collection.GenTraversableOnce[?]
[error]           |  ${nodes.toList.flatMap(_.dotString.lines).mkString(System.lineSeparator() + "  ")}
[error]                                                 ^
[error] /code/wom/src/main/scala/wom/views/GraphPrint.scala:168:48: type mismatch;
[error]  found   : java.util.stream.Stream[String]
[error]  required: scala.collection.GenTraversableOnce[?]
[error]          |  ${nodes.toList.flatMap(_.dotString.lines).mkString(System.lineSeparator() + "  ")}
[error]                                                ^
[error] four errors found

Any suggestion?

[mcovarr]

@wdesouza I am seeing this as well. This fork was created just before #6194 that upgraded Cromwell's Java version from 8 to 11. I think these compilation errors may represent some (hopefully minor) incompatibilities.

[henriqueribeiro]

@mcovarr @wdesouza I will try to update this pr asap

[wdesouza]

I've changed _.dotString.lines to _.dotString.linesIterator and it worked. Thanks @henriqueribeiro and @mcovarr

[geertvandeweyer]

Hi @henriqueribeiro ,

Do you think I can compile this branch with functionality of the latest cromwell release (77) ? I'm now using the branch based on cromwell 58, but more recent versions have retry strategy that's interesting as well:

• you retry logic handles spot kills
• cromwell retry handles the fetch_and_run is a directory problem.

=> both would be great, but since it doesn't get approved, I hope to make a new custom build. However, it says here there are conflicts...

Greetings, geert

[henriqueribeiro]

Hey @geertvandeweyer. I'm preparing a new release with more functionalities based on cromwell 78. It should be ready in the next few days.

Are you in the cromwell slack?