how to add --privilleged for docker when run cromwell in aws

[openbioinfo]

how to add --privilleged for docker when run cromwell in aws

where to add this option ?

[markjschreiber]

There’s currently no way to do this. It would need to be added to the part of the code that constructs the job definition.

What is the scenario requiring running as privileged?

On Thu, Sep 17, 2020 at 4:53 AM openbioinfomatics for more people who need it [email protected] wrote:

how to add --privilleged for docker when run cromwell in aws

where to add this option ?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/broadinstitute/cromwell/issues/5863, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2E6ELZ4NI2CUEDCRPJHVDSGHFCJANCNFSM4RQDAKPQ .

[openbioinfo]

i plan to run singularity images in docker container.

[guma44]

Did you try to implement your own docker-submit in the config file?

[openbioinfo]

no. in aws mode, i dont find docker-submit.

i prefer --privileged=true as default for cromwell with aws as backend.

[guma44]

"submit-docker" (sorry for reversal) is one of the configuration option in Cromwell config file. See eg. here how additional volumes are mounted (last section): https://davetang.org/muse/2019/12/24/execute-gatk-workflows-locally. In the same way, you can run docker command that passes --privileged=true option.

[openbioinfo]

thanks for you reply.

i mean in aws backend mode, instead of local mode. there is no option to set submit-docker, i attached the backend part of my aws.conf as follows.

backend {
     default = "AWSBATCH"
     providers {
         AWSBATCH {
             actor-factory = "cromwell.backend.impl.aws.AwsBatchBackendLifecycleActorFactory"
             config {
                 // Base bucket for workflow executions
                 root = "s3://yuce/cromwell-execution"
                 // A reference to an auth defined in the `aws` stanza at the top. This auth is used to create
                 // Jobs and manipulate auth JSONs.
                 auth = "default"

                 numSubmitAttempts = 3
                 numCreateDefinitionAttempts = 3

                 concurrent-job-limit = 16

                 default-runtime-attributes {
                    queueArn: "arn:aws-cn:batch:cn-northwest-1:723230375162:job-queue/first-run-job-queue",
                 }

                 filesystems {
                     s3 {
                         // A reference to a potentially different auth for manipulating files via engine functions.
                         auth = "default"
                     }
                 }
             }
         }
     }
}

[guma44]

By "no option to set" do you mean that is not allowed to do this? Cannot you just add this section? Local is just an example. I have UGE backend where I override "submit-docker" to use it with singularity.

[openbioinfo]

yes. i hava tried. in aws mode, cromwell does not support submit-docker.

https://github.com/broadinstitute/cromwell/issues/5863#issuecomment-699685666

There’s currently no way to do this. It would need to be added to the part of the code that constructs the job definition. What is the scenario requiring running as privileged? … On Thu, Sep 17, 2020 at 4:53 AM openbioinfomatics for more people who need it @.***> wrote: how to add --privilleged for docker when run cromwell in aws where to add this option ? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5863>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2E6ELZ4NI2CUEDCRPJHVDSGHFCJANCNFSM4RQDAKPQ .

[guma44]

Ah, OK. This looks like something to add.

[scanon]

We have a similar need. This overlaps a little with #4579. It would be useful if the submit-docker was parameterized similar to how it is for some of the other backends.

[markjschreiber]

Thanks for the feedback. Can you elaborate more on the need to be able to run a container as privileged?

It could (in theory) be parameterized if required but it seems hazardous to have this be the default.

On Tue, Jan 5, 2021 at 5:42 PM Shane Canon [email protected] wrote:

We have a similar need. This overlaps a little with #4579 https://github.com/broadinstitute/cromwell/issues/4579. It would be useful if the submit-docker was parameterized similar to how it is for some of the other backends.

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/broadinstitute/cromwell/issues/5863#issuecomment-754946394, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF2E6ELZQ5HVOSV2JRMIH3LSYOIVRANCNFSM4RQDAKPQ .

[leipzig]

One scenario that comes to mind is running anything that requires a docker or singularity server, such as Nextflow. So if you need to run Nextflow as a step in a larger WDL workflow it is going to want to pull images