psutil
psutil copied to clipboard
giampaolo
Expose cgroup as process attr
From [email protected] on March 30, 2014 05:12:36 Original issue: http://code.google.com/p/psutil/issues/detail?id=490
On linux, cgroup usage is increasingly common (systemd, lxc, docker), it would be nice to have that exposed on a process as an attribute.
From g.rodola on April 07, 2014 09:28:49
I've been meaning to look into this for a while now but I've never really had the chance to look into how Linux cgroups work and how to configure them on a per-process basis (yet). It must be noted that Process.rlimit() was introduced as some kind of substitute for Linux cgroups as it allows to configure many aspects of the process execution with a clean(er) API: https://pythonhosted.org/psutil/#psutil.Process.rlimit If there's one thing I understood from Linux cgroups is that it's not immediately clear how to configure them, therefore it's also not clear what kind of API we should expose. I'm kind of a newbie here so any kind of suggestion is appreciated.
From [email protected] on April 07, 2014 09:54:48
so configuring cgroups i think is probably out of scope, things there are still in a bit of flux (direct sys fs mods to arbiter either in the form of cgmanager or systemd accessed via debus).
exposing the details of a process's cgroup those is relative straightforward though http://blog.docker.io/2013/10/gathering-lxc-docker-containers-metrics/ ideally to me is filter on querying processes by cgroup name, with each process obj having a link/attr to its cgroup which is an object that exposes the overall cgroup stats (mem, cpu usage, etc) which makes collecting container/process group stats trivial.
@giampaolo - Are you open to PRs for resolving this? (Asking since you rejected the PR for #765) It will have to be linux specific (since cgroups are a linux construct), and I guess it will use the libcgroups python bindings internally.
One possible use case (out of many) would be to discover how much RAM is REALLY available for my process. When running in a containerized/cgroup environment, the general memory usage/limits (which are discovered internally using sysinfo) are often incorrect.
A good write up on cgroups and their influence on memory limits can be found here: http://fabiokung.com/2014/03/13/memory-inside-linux-containers/
The main problem I have with cgroups is that I don't use/know them and it looks like a pretty extended subject. Before a PR I would like to see a discussion about what capabilities should be implemented and what the API should look like.
The main question is: should psutil be a fully featured cgroups library? My hunch is "no", because that would be too much and outside of the scope of psutil as a project.
For instance, the API of libs like this one: https://github.com/cloudsigma/cgroupspy ...gives me the sensation that this cgroup thing is something big which deserves a separate, dedicated project. I don't want something like that in psutil.
On the other hand, something like this:
https://github.com/francisbouvier/cgroups
...intrigues me because it seems it could be relatively easy to associate a cgroup to a PID, and that looks like something which could be integrated in psutil.Process class somehow.
As to how to do that in terms of API, I'm not sure, and that's because I'm ignorant about how cgroups work exactly. Judging from https://github.com/francisbouvier/cgroups it seems they are bound to a name: you first create a cgroup, then you associate a PID to it. That's kind of a bummer because it requires a separate Cgroup(name) class of some sort.
That's why I'm calling for others who are more familiar than me with cgroups to think about a simple API and start a discussion.
If this were to land in psutil it's likely gonna be a read-only method returning a dict. Something like:
@wrap_exceptions
def cgroups(self):
with open_binary("%s/%s/cgroup" % (self._procfs_path, self.pid)) as f:
cgroups = {}
for line in f:
line = line.strip()
cid, subsys, name = line.split(':', 2)
if name.startswith('/'):
name = name[1:]
if subsys == "name=systemd" and cid == "1":
subsys = "systemd"
if name:
cgroups[subsys] = name
return cgroups
I wonder why systemd gets special treatment in such proposition? cgroup names are arbitrary, there is no "subsys". Just systemd uses such naming.
In my case, on non systemd in all cases "subsys" is empty like:
{'': 'rsnapshots/db'}
{'': 'rsnapshots/nonroot'}
{'': 'rsnapshots/misc'}
etc.
Anyway +1 for exposing cgroup information.
Hi, thanks for an awesome library. Any news or new ideas regarding this?
I'm afraid I don't know enough about Linux cgroups to design this feature. For instance: should we have a read-only Process.cgroups() method returning a dict or also allow it to set limits as in Process.cgroups(memory_limit=1000_000_000)? What are all those limits (memory, cpu, disk i/o, ...)?
Somebody with cgroups experience should chime in. Taking a look at how other third-party libs exposed this in terms of API would also be beneficial.
I think it can be very useful to know if there are any limits placed on a process via cgroups, because this is how containers (for example with Docker) and high-performance computing clusters limit resource usage of programs.
I have written an implementation for the memory limit here https://github.com/HALFpipe/HALFpipe/blob/main/halfpipe/memory.py#L25-L163, and detecting the CPU limit should be similar. Please let me know if you would be interested in me preparing a pull request.
For reference, there is a very well crafted and complete implementation of cgroup stats and whatnot at https://github.com/peo3/cgroup-utils
Linux cgroup information for memory management is more than common in cloud environment, given Kubernetes are popular these days. Running code inside a container is almost certain for all cloud deployments. Knowing the physical memory max and usage helps nothing but knowing cgroup-level memory max/usage does.
In my scenario I have a machine-learning code running in Kubernetes on AWS. The code reads as much data as possible by measuring current memory usage percentage out of max. It works well with psutil on phyiscal machine, but doesn't work at all in a containerized environment due to that it has no knowledge about cgroup level memory usage.
