reparenting tasks, from docker to mesos, does not work with systemd

[jdef]

for context: https://github.com/mesosphere/universe/pull/200#issuecomment-135461209

given the current state of the integration:

• additional mesos work is likely required
• additional docker work is likely required

it's possible that the upcoming unified containerizer could resolve this as well.

[jdef]

talked this around internally:

• we can't rely on people having a particular version of docker installed
• nor can we rely on changing docker command line args for specific exec driver options (e.g. to force use of cgroups driver instead of systemd)

basically we're dependent upon mesos implementing better support for systemd:

• https://issues.apache.org/jira/browse/MESOS-3007

[ravilr]

i tested this out with the build from git master without --contain-pod-resources on a RHEL system using systemd for launching docker-1.7.1, mesos-slave. the pod containers fail to come up as docker daemon complains about 'invalid args', most likely due to cgroup_parent parameter being passed to docker run.

With --exec-opt native.cgroupdriver=cgroupfs option to docker daemon, the pod containers come up fine. and all docker containers cgroup moved under resource_container/docker_container_id.

[jdef]

Thanks for testing that out!

On Fri, Sep 4, 2015 at 5:26 PM, ravilr [email protected] wrote:

i tested this out with the build from git master without --contain-pod-resources on a RHEL system using systemd for launching docker-1.7.1, mesos-slave. the pod containers fail to come up as docker daemon complains about 'invalid args', most likely due to cgroup_parent parameter being passed to docker run.

With --exec-opt native.cgroupdriver=cgroupfs option to docker daemon, the pod containers come up fine. and all docker containers cgroup moved under resource_container/docker_container_id.

— Reply to this email directly or view it on GitHub https://github.com/mesosphere/kubernetes-mesos/issues/460#issuecomment-137856933 .

[ravilr]

@jdef http://mesos.apache.org/blog/mesos-0-27-1-released/ refers to systemd support issue in mesos as closed. Does it solve this issue in k8sm?

[jdef]

@ravilr thanks for posting. unfortunately this problem is not yet solved. it stems from how containers are managed in mesos vs. how they're managed by kubernets/docker. the existing k8s/docker integration is completely unaware of mesos containerization and vice-versa: mesos is completely unaware of the fact that k8s/docker is spawning new containers outside of the mesos control domain.

we've discussed building out support for a k8s runtime implementation that's compatible with mesos native containerization (especially given the upcoming improvements related to the unified image provisioning in mesos). that work may come down the road a bit.

HTH

[ravilr]

thanks for the explanation. the changes in https://issues.apache.org/jira/browse/MESOS-3425 seems to assign the executor process to a newly created systemd slice: 'mesos_executors.slice' . if the docker_cgroup_parent is also set to this cgroup heirarchy (/sys/fs/cgroup/systemd/mesos_executors.slice), all the docker containers would also be contained in the slice right?

Also, this was opened today in upstream: https://github.com/kubernetes/kubernetes/issues/22964

[jdef]

Last time I tried to use docker's flag-based cgroup reparenting things broken horribly: docker made some pretty big assumptions about systemd cgroup naming that are not compatible w/ mesos. i haven't checked whether this has been fixed in recent docker releases.

On Tue, Mar 15, 2016 at 2:56 PM, ravilr [email protected] wrote:

thanks for the explanation. the changes in https://issues.apache.org/jira/browse/MESOS-3425 seems to assign the executor process to a newly created systemd slice: 'mesos_executors.slice' . if the docker_cgroup_parent is also set to this cgroup heirarchy (/sys/fs/cgroup/systemd/mesos_executors.slice), all the docker containers would also be contained in the slice right?

Also, this was opened today in upstream: kubernetes/kubernetes#22964 https://github.com/kubernetes/kubernetes/issues/22964

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/mesosphere/kubernetes-mesos/issues/460#issuecomment-196972413