tmpnb
tmpnb copied to clipboard
jupyter
Allow for bursting CPUs when less users are active
When not many people are on a tmpnb installation, subdivide cpu_shares amongst them. I'd like for it to be that when only one user is on a box, they get access to all the cores. When more active users join in, divide up the CPU shares.
This can be done, but is a bit outside of Docker itself. If we were using systemd, we could set cgroups properties on individual containers:
systemctl set-property docker-4be96b853089bc6044b29cb873cac460b429cfcbdd0e877c0868eb2a901dbf80.scope CPUShares=512
Otherwise, we can interact directly with the cgroups filesystem to handle this. A little bird tells me that the major distributions are aligning with systemd in the future, so this could get easier.
Just what I needed, a list of distros with systemd enabled by default.
Booted an Ubuntu 14.10 server just in case Wikipedia was out of date. Systemd and systemctl is set up, but the image I ran from booted with init instead:
root@ub1410:~# systemctl
Failed to get D-Bus connection: No connection to service manager.
root@ub1410:~# systemd
Trying to run as user instance, but the system has not been booted with systemd.
root@ub1410:~# ps -p 1 -o comm=
init
CentOS, Fedora, Debian (Jessie - testing, Sid - unstable), OpenSUSE are all set to go and benefit from the ease of working with this. CoreOS too of course, but that's a whole new rabbit hole...
I wonder if other projects have already scoped the dynamic resizing out.
Doing this directly isn't terrible, but it is a bit murky. I used a posting on the Docker mailing list from @jpetazzo about changing the cgroups settings directly.
The only difference here is that lxc got replaced with docker from that posting:
echo "1024" > /sys/fs/cgroup/cpu/docker/bb4816c2099397b09ae4e16ecc5fa63d663799ee1968cc9b1b084f0d733d9428/cpu.shares
Since we have the container id on the proxy, we could monitor it externally or do it as part of our current heartbeat.
What would people think of just declaring the max_cpus, max_memory (or detecting it) and the pool_size, then scaling individual containers automatically?
/cc @smashwilson @roalexan @dciborow @koldunovn
:+1: Sounds excellent. I would add max_cpus and max_memory to the argument list, and probe for them if they aren't specified; that lets you manually crank them down if you want to, but keeps the list of required arguments small.
Is the /sys/fs path consistent across host OS distributions? Hardcoding it like that seems fragile to me. It would probably be best to feature-flag the whole behavior, so it doesn't outright break on systems that differ slightly.
Is the /sys/fs path consistent across host OS distributions?
No idea, I recognize this is somewhat of a disastrous idea if we don't have something cross-linux-distribution.
Actually, it might very well be consistent across modern hosts that we're aiming for:
https://docs.docker.com/articles/runmetrics/
Alright, I've dug into this cgroupspy module built by cloudsigma. They've made some nice utility functions that let you change the cgroups filesystem path as need be (so you can handle legacy paths). The defaults are exactly what you see on modern Ubuntu versions (14.04, 14.10). I didn't try other distributions though.
In [1]: import cgroupspy
In [2]: import cgroupspy.trees
In [3]: gt = cgroupspy.trees.GroupedTree()
In [4]: container_id = 'ebbd92933463cf9fd3953a0b347cfdaf6acab919d296c2f65da602886be278b2'
In [5]: node = gt.get_node_by_name(container_id)
In [6]: node.cpu.shares
Out[6]: 16
In [7]: node.cpu.shares = 64
In [8]: node.cpu.shares
Out[8]: 64
In [9]: !cat /sys/fs/cgroup/cpu/docker/$container_id/cpu.shares
64