Deprecated

Please note that the CGManager project has been deprecated in favor of using the kernel's CGroup Namespace or lxcfs' simulated cgroupfs.

See https://s3hh.wordpress.com/2016/06/18/whither-cgmanager/ for details.

=== Intro ===

This is a motivation, description and explanation of the cgmanager design. The original design RFC was described here:

http://lwn.net/Articles/575672/ http://lwn.net/Articles/575683/

And much of it still holds (and is cut-pasted, though edited, here).

=== Cgmanager Design ===

One of the driving goals is to enable nested lxc as simply and safely as possible. If this project is a success, then a large chunk of code can be removed from lxc. I'm considering this project a part of the larger lxc project, but given how central it is to systems management that doesn't mean that I'll consider anyone else's needs as less important than our own.

This document consists of two parts. The first describes how I intend the daemon (cgmanager) to be structured and how it will enforce the safety requirements. The second describes the commands which clients will be able to send to the manager. The list of controller keys which can be set is very incomplete at this point, serving mainly to show the approach I was thinking of taking.

=== Summary ===

Each 'host' (identified by a separate instance of the linux kernel) has exactly one running daemon to manage control groups. This daemon answers cgroup management requests over a dbus socket, located at /sys/fs/cgroup/cgmanager/sock. The /sys/fs/cgroup/cgmanager directory can be bind-mounted into various containers, so that one daemon can support the whole system. (Bind-mounting the directory rather than the socket itself allows a container to proceed if the cgmanager is restarted, creating a new socket.)

Outline: . A single manager, cgmanager, is started on the host, very early during boot. It has very few dependencies, and requires only /proc, /run, and /sys to be mounted, with /etc ro. It mounts the cgroup hierarchies in a private namespace and set defaults for clone_children and use_hierarchy. It opens a Unix socket at /sys/fs/cgroup/cgmanager/sock. . A client (requestor 'r') can make cgroup requests over /sys/fs/cgroup/cgmanager/sock using dbus calls. Detailed privilege requirements for r are listed below. . The client request will pertain an existing or new cgroup A. r's privilege over the cgroup must be checked. r is said to have privilege over A if A is owned by r's uid, or if A's owner is mapped into r's user namespace, and r is root in that user namespace. . The client request may pertain a victim task v, which may be moved to a new cgroup. In that case r's privilege over both the cgroup and v must be checked. r is said to have privilege over v if v is mapped in r's pid namespace, v's uid is mapped into r's user ns, and r is root in its userns. Or if r and v have the same uid and v is mapped in r's pid namespace. . r's credentials will be taken from socket's peercred, ensuring that pid and uid are translated. . A request to chown a cgroup requires a uid U and gid G. . If r is in the same pid and user namespaces as the cgmanager, then v, U and G can be passed as integer arguments over the D-Bus requests. . If r is not in the same namespaces as the cgmanager, then V, U and G must be passed as SCM_CREDENTIALs so that the cgmanager receives the translated global pid/uid/gid. Since D-Bus does not support sending SCM_CREDENTIALs as part of a D-Bus message, the D-Bus arguments include a file descriptor. The SCM_CREDENTIALs are sent over the file descriptor after the D-Bus transaction completes, and the final result is sent over the same file descriptor. . It is desirable that all transactions can be accomplished with simple D-Bus transactions. Therefore a cgroup manager proxy (cgproxy) is provided. This will move /sys/fs/cgroup/cgmanager to /sys/fs/cgroup/cgmanager.lower, then serve as a proxy translating D-Bus requests received on /sys/fs/cgroup/cgmanager/sock into SCM-enhanced D-Bus requests on /sys/fs/cgmanager/cgmanager.lower/sock. . In plain D-Bus transactions, the requestor r's credentials are read from the socket. . In SCM-enhanced D-Bus transactions, the proxy p's credentials are read from the socket. The requestor's credential is sent as an SCM_CREDENTIAL.

Privilege requirements by action: * Requestor of an action (r) over a socket may only make changes to cgroups over which it has privilege. * Requestors may be limited to a certain #/depth of cgroups (to limit memory usage). This is not yet implemented. * Cgroup hierarchy is responsible for resource limits. To this end, a request to chown cgroup A to uid U will only chown the directory itself (allowing child cgroup creation) and the tasks and cgroup.procs file. * A requestor must either be uid 0 in its userns with victim mapped ito its userns, or the same uid and in same/ancestor pidns as the victim * If r requests creation of cgroup '/x', /x will be interpreted as relative to r's cgroup. r cannot make changes to cgroups not under its own current cgroup. * Root in the cgmanager's pid namespace may 'escape' to the cgmanager's cgroup with a special MovePidAbs command. * A proxy may move a task over which it has privilege to the proxy's own cgroup. This allows the proxy to mimic the cgmanager's special root-may-escape semantics in its own container. * If r requests creation of cgroup '/x', it must have write access to its own cgroup. * if r requests setting a limit under /x, then . either r must be root in its own userns, and UID(/x) be mapped into its userns, or else UID(r) == UID(/x) . /x must not be / (not strictly necessary, all users know to ensure an extra cgroup layer above '/') . setns(UIDNS(r)) would not work, due to in-kernel capable() checks which won't be satisfied. Therefore we'll need to do privilege checks ourselves, then perform the write as the host root user. (see devices.allow/deny). Further we need to support older kernels which don't support setns for pid.

Types of requests:

• r requests creating cgroup A'/A . lmctfy/cli/commands/create.cc . Verify that UID(r) mapped to 0 in r's userns . R=cgroup_of(r) . Verify that UID(R) is mapped into r's userns . Create R/A'/A . chown R/A'/A to UID(r)
• r requests to move task x to cgroup A. . lmctfy/cli/commands/enter.cc . r must send PID(x) as ancillary message . Verify that UID(r) mapped to 0 in r's userns, and UID(x) is mapped into that userns (is it safe to allow if UID(x) == UID(r))? . R=cgroup_of(r) . Verify that R/A is owned by UID(r) or UID(x)? (not sure that's needed) . echo PID(x) >> /R/A/tasks
• r requests chown of cgroup A to uid X . X is passed in ancillary message
  • ensures it is valid in r's userns
  • maps the userid to host for us . Verify that UID(r) mapped to 0 in r's userns . R=cgroup_of(r) . Chown R/A to X
• r requests cgroup A's 'property=value' . Verify that either
  • A != ''
  • UID(r) == 0 on host In other words, r in a userns may not set root cgroup settings. . Verify that UID(r) mapped to 0 in r's userns . R=cgroup_of(r) . Set property=value for R/A
  • Expect kernel to guarantee hierarchical constraints
• r requests deletion of cgroup A . lmctfy/cli/commands/destroy.cc (without -f) . same requirements as setting 'property=value'
• r requests purge of cgroup A . lmctfy/cli/commands/destroy.cc (with -f) . same requirements as setting 'property=value'

Long-term we will want the cgroup manager to become more intelligent - to place its own limits on clients, to address cpu and device hotplug, etc. Since we will not be doing that in the first prototype, the daemon will not keep any state about the clients.

=== Another look at the safety of requests ===

Notes:

1. In a plain D-Bus call, the proxy is the requestor.
2. If a client does an SCM call to the cgmanager socket, then the proxy is the requestor.
3. In any call over a proxy, the proxy won't be able to make changes outside its own cgroups. If it misbehaves, damage is contained so it only damages itself..
4. Chained proxying is not supported. If a proxy gets a request where proxy != requestor, the call is rejected.
5. The identity of the proxy (which may be the requestor) cannot be forged; it is taken from the socket credential. A more privileged user must not allow a less privileged task to have access to the opened DBus socket, as the credential will be that at the time of connect().

On newer kernels, cgmanager can tell whether a proxy or requestor is in the same namespace as itself. On older kernels, it cannot.

. for Create, this is ok. We have the proxy's real pid and can constrain create under its cgroup. . for getPidCgroup, we can ensure that only results under the parent's cgroup are returned. we can NOT ensure that results will make sense for plain DBus calls, as we cannot guarantee that proxy is in the same ns as cgmanager. However, this is not unsafe. When we can and do detect that p is in a different pid namespace, then we reject the call, because the result cannot be sensible. . for chmod: We constrain under proxy's cgroup, so this is safe. . for chown: on older kernel we cannot guarantee that the uid/gid make sense on the host; However . root on the root host - no translation necessary . root in a non-user-ns container: no translation necessary . root in a unprivileged container: won't have privilege to do any chown without going through a proxy. Therefore rejecting calls from another namespace is not necessary. The worst it will do is to give -EPERM for calls which for root in a unprivileged container otherwise would be allowed to do. . movepid: . root on root host - fine . root in a non-user-ns container: we can only ensure that the victim be under the proxy's cgroup. If that is the case, then root (which is also root on the host) is allowed to move the task. When we can and do detect a different pid namespace, then we reject the call because the results cannot make sense. . MovePidAbs: On an older kernel, or if the task is in a different namespace, then this requires a proxy. The cgmanager will only allow escaping up to the level of the proxy. . root on root host - allowed to escape. . root in a non-user-ns container: allowed to escape up to the proxy's level. If the host misconfigures the container so that the host's proxy is in the container, then root can escape completely. . if root tries to mimick a proxy, then it can only escape to the proxy's level - it's own. So it cannot escape at all.