Add support --privileged when create service

[jimmyxian]

When create some container, we need privileged=True. But swarmkit does not support this. I want to implement this by the following steps:

• Add privileged field in ContainerSpec.
• Pass privileged to HostConfig when create container.

If it's the right direction, I will submit a PR. :)

Also, there so many container options in docker-engine. If all the options are supported in that way, It's a huge work.Maybe we need have a good way to support all the options?

[dmcgowan]

Privileged containers are needed for any use case which needs to spin up Docker-in-Docker, a common pattern for testing not only docker itself but complex build setups such as allowed by https://github.com/docker/golem. I am looking forward to porting golem to swarm when this is enabled.

[stevvooe]

Also, there so many container options in docker-engine. If all the options are supported in that way, It's a huge work.Maybe we need have a good way to support all the options?

We don't actually want to do this. There are number of things that are hard or impossible to support in a clustered environment. There are other things that are downright insecure, such as privileged, that need to be rethought.

What aspect of privileged containers are you using? For example, what part enables DIND to work?

[dmcgowan]

We don't actually want to do this.

Many people will and it is a common pattern. However I do agree that time should be spent to figure out how to do it right since adding this will mean it can never be removed along with any security headaches it brings.

As for DIND in particular. I am not sure of all the settings that would need to be added to enable this, getting this working I think would cover many use cases for CI. Try running the make test run command without the privileged flag will yield...

mount: permission denied
Could not mount /sys/kernel/security.
AppArmor detection and --privileged mode might break.
mount: permission denied

The reason is obvious from https://github.com/docker/docker/blob/master/hack/dind.

[stevvooe]

@dmcgowan Supporting all of the container and host config options levied a massive amount of complexity on the swarm project for little gain. Each feature we add in services needs to be well-considered and designed for operation in a clustered environment. This is especially true for anything that requires privileged execution. (interesting aspect: ContainerSpec ends up looking a lot like ~~HostConfig~~ container.Config, in an image).

Right now, swarm services aren't really setup well for CI or build use cases. We don't really have the concept of batch jobs and collecting logs is problematic. This will come in the future.

Do we have any online use cases for actual services?

[dmcgowan]

Looking at the container setup code within docker the following is done by setting the privileged flag

• Set empty process label
• Warn of incompatibility with user namespaces
• Add all host devices from /dev
• Add device cgroup access rwm allow, see https://github.com/docker/docker/blob/master/daemon/oci_linux.go#L92
• Add all capabilities
• Clear read only flag for /sys mount
• Set read only paths to nil (*specs.Spec).Linux.ReadonlyPaths = nil, see https://github.com/docker/docker/blob/master/daemon/oci_linux.go#L561
• Set masked paths to nil (*specs.Spec).Linux.MaskedPaths = nil, see https://github.com/docker/docker/blob/master/daemon/oci_linux.go#L562
• Clear read only flag for cgroup mount
• Set app armor profile "unconfined"

[stevvooe]

@dmcgowan Wow, thanks for the overview!

cc @aluzzardi

[chanwit]

Global service fits really well for deploying monitoring agents on every node. And most of those agents require privileged access to some /proc. In my case, I want to monitor conntrack.

Please add support for this privileged flag, cc @aluzzardi

[stevvooe]

@chanwit: when not running with --privileged, what error do you get when trying to monitor conntrack?

[aluzzardi]

@stevvooe @chanwit @jimmyxian I was initially against adding --privileged. Seeing how many requests we got for this, I'm in favor for adding it now

[jimmyxian]

@aluzzardi Thanks, I will rebase the PR. :)

[chanwit]

@stevvooe my bad. it's another issue :-( not relating to privileged.

Anyway I gave up running monitoring agents inside container for now.

[sashkachan]

Trying to run weave-scope using the new swarm, also hitting the roadblock with the privileged mode (or the lack thereof). Would be great to have it supported.

[lukemarsden]

Any update on this @aluzzardi? Would be great to get weave scope working with swarmkit. And we really do need to access /proc. cc @errordeveloper

[errordeveloper]

@alex-glv we have hack that makes it work until this gets resolved: https://github.com/weaveworks/scope-global-swarm-service.

Also, there is a conversation around explicit --capabilities flag: https://github.com/docker/docker/pull/26849#issuecomment-252704844.

[thaJeztah]

Also, related issue in docker/docker https://github.com/docker/docker/issues/24862

[alexellis]

Where are we with this issue now? @aluzzardi - still feeling positive?

[marcellodesales]

:( I need to run Global services please!

[stevvooe]

Note that we are attempting to address this with security profiles, proposed in #1722. The goal is to provide cluster operators much finer grain of access control for security containment without making the model more complex that --privileged.

[shankarkc]

Is there an ETA for this? I am hitting small /dev/shm on docker containers. I have to increase the size. I have used service to create containers. Now as service doesnt allow me to run in privilaged mode, I cannot unmount/remount /dev/shm

[thaJeztah]

@shankarkc would mounting a tmpfs work? Haven't tried, but;

--mount type=tmpfs,target=/dev/shm,.....

Possibly requires docker 1.13 (for tmpfs)

[shankarkc]

Thanks for input. Installed 1.3 It works. On Nov 23, 2016 9:17 PM, "Sebastiaan van Stijn" [email protected] wrote:

@shankarkc https://github.com/shankarkc would mounting a tmpfs work? Haven't tried, but;

--mount type=tmpfs,target=/dev/shm,.....

Possibly requires docker 1.13 (for tmpfs)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/docker/swarmkit/issues/1030#issuecomment-262551740, or mute the thread https://github.com/notifications/unsubscribe-auth/AEb7VwTJd6gbf-x7o2Cf5vV8NEGTbMGyks5rBF_zgaJpZM4I5UR9 .

[ginjo]

Curious where we are on the issue of allowing deeper privileges in swarm containers. My use case involves vpn software that requires addition of an ip interface and a couple of routing rules.

Is the focus of this issue now on security profiles https://github.com/docker/swarmkit/pull/1722 , and if so, will they allow deeper privileges in swarm containers?

[mikeytag]

I second @ginjo. I'm trying to get an OpenVPN container up in swarm but can't without privileged, cap_add=NET_ADMIN and device=/dev/net/tun

If anyone has a workaround. I'm all ears.

[marcellodesales]

I have a couple of use cases for the need of running mode=global containers in my cluster:

• We are still using older Monitoring images that require privileges...
• I have a raspberry pi cluster that accesses GPIO that require privileges...

Any status?

[sono-bfio]

👍 This or the at least the ability to pass in devices in my cases (Working with nvidia GPUs in my case).

[AkihiroSuda]

Until https://github.com/docker/swarmkit/pull/1722 gets merged, a workaround is to create a service with bind-mounting the API socket, and invoke docker run --privileged within a service container.

i.e. docker service create --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock foo docker run --rm --privileged bar baz

[ventz]

Same use case as @mikeytag -- need to add OpenVPN as a service.

[zerowebcorp]

@AkihiroSuda Would you please elaborate how this can be done ?

[AkihiroSuda]

The command I mentioned above is composed of two parts:

1. docker service create --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock foo
2. Execute docker run --privileged in the service container created in the step 1

The step 1 creates a service with the access to Docker API socket. So you need to install the docker binary (only client is enough) to the image foo.

In the step 2, you can create privileged containers using the socket.

Note that the privileged programs still does not work on the service. You can just let the service create new containers for the privileged programs. So you might need some additional work about the lifecycle of the containers created.

[alexellis]

@getvivekv

At best this is a temporary work-around ... at worst it's confusing and as @AkihiroSuda mentions it opens a can of worms around managing the privileged containers created. You may be better off with creating your privileged containers manually and then using an attachable overlay network so that you can join them to swarm services.

I.e.

• Create an attachable network
• Deploy services onto the swarm
• Create ad-hoc containers with --network flag

$ docker network create mixed --driver overlay --attachable
c9xwtgk8259j8rnq3xifvt9kv
$ docker service create redis --network=mixed redis
$ docker run --network=mixed -d some_privileged_image_that_uses_redis

+1 x 1000 for --privileged services on Swarm, for Raspberry Pi etc.

Cross-referencing:

• (Missing from Swarmmode --cap-add #25885)

https://github.com/docker/docker/issues/25885

• api: add support for security profiles #1722

https://github.com/docker/swarmkit/pull/1722