Filesystem quotas

[fabiokung]

Following up the discussion from the last containerd summit:

https://github.com/containerd/containerd/blob/6ae973362af4b9ae27fb335efba6b5e508bc1752/docs/dockercon-summit.md#discussion-points

How to support per-container fs quotas? The implementation of quotas (usually called project quotas) is tied to the specific implementation of the Snapshotter being used.

• What is the right abstraction to support different project quota implementations (brtfs, overlay+xfs, overlay+ext4, and maybe in the future zfs, devicemapper and others since anyone can implement Snapshotter)?
• What is the right granularity for quotas? Some project quota implementations allow volumes and container writeable layers to be in the same quota group, and share the same quota limit. Is it something that need to be supported, or volumes will always be in different quota groups, managed completely outside containerd?
• During the discussion, encouraging read-only container filesystems was mentioned, and forcing people to use volumes for R/W directories, with quotas being managed by external volume managers. I agree this would be ideal, but is it even possible to force read-only rootfs for everyone?

[Random-Liu]

/cc @vishh

[cpuguy83]

It could be helpful to get at the actual project quota from outside the snapshotter, but I can see this being a mess, especially when the "volume" is a different filesystem than the fs the snapshotter is running on.

[vishh]

The goal of not needing a rw writable layer is ideal but has been a pipe dream thus far. I'd recommend not conflating volumes which is external to containerd with writable layer that is internal. Since containerd manages the writable layer and does not allow introspecting the writable layer by external entities, it needs to provide a means to track and limit usage ideally using Linux Quota.

[AkihiroSuda]

Inevitably, image builder (e.g. docker build) requires RW rootfs. So I'm +1 on quota (as in current docker overlay2 on xfs )

[yichengliu58]

Hi, I'm facing this problem at the moment, that I need to set the size of container's roofs but haven't found a way.

I know docker supports a config item called --storage-opt dm.basesize which can be used to set rootfs size of a container, because dockerd actually uses containerd as its backend, I'm curious why can't containerd support it?

So are there any update? Is there a config item can be used to set rootfs size now?

[sharego]

when using devmapper the base_image_size setting all container rootfs size, which need to be control per-container too. I tried do this by add a special label when create a container: https://github.com/sharego/containerd/commit/5e58aac9e19d4e7f54aface8159bd235d8f3133f. A solution for all snapshotter is great.

[AkihiroSuda]

Moved to milestone 1.6

[lining2020x]

Hello guys, will this feature be included in the future version of containerd?

[dashjay]

When using the built-in storage control of K8S, the POD will be expelled when the disk usage of the POD exceeds the limit. This does not what we desired. but I things are difficult to handle:

• there are a lot of filesystem: xfs, nfs, overlayfs..... we can not do the same thing for every fs.
• maybe k8s will do this, so we don't have to do it again (maybe)
• ...... a lot ...

[zheng199512]

How is it going now？

[timchenxiaoyu]

wait 1.6 ？

[AkihiroSuda]

Linking PR https://github.com/containerd/containerd/pull/5859

[lining2020x]

Hello guys, will this feature be included in the future version of containerd?

Again, thanks.

[zvier]

We set xfs quota to limit rw layer disk usage with use dockerd as container.

{
  "storage-driver": "overlay2",
  "storage-opts":[
    "overlay2.override_kernel_check=true",
    "overlay2.size=50G"
  ],
...

But when consider switch to containerd, this feature will lost. It seems that this feature is also needed for containerd.

[vtrenton]

any update on this issue? fs quotas would be very useful for some workloads that tend to write a lot to tmpfs. I dont see anything merged into 1.6 around this. maybe i'm missing something?

[pacoxu]

/assign I may work on this recently.

[pacoxu]

Sync some info from the WIP PR https://github.com/containerd/containerd/pull/5859#issuecomment-1980294951 and raise some questions for discussions around this feature. /cc @yylt

I am not sure about the future of this feature. But we have similar requirements as well, users want a similar behavior like docker in this case. (limit the disk instead of evicting the pod directly)

• From kubernetes perspective, pod will be evicted if pod disk usage reached the limit in empty dir.

Questions we need to discuss about this feature:

1. Collect clearer user stories: as I know, some users want a limited pod disk and the pod can work even it reached the limit.(something like a pod running in maintenance mode, and also avoid making node disk in pressure.) Some actions like backup/archive/save-checkpoint are needed before removing the pod with full disk.
2. Should we support it in CRI? Then kubelet can use this feature to define the quota later. It can be a configuration per node or per pod, or even per container.
   • Is this a containerd configuration? Will CRI overwrite the configuration?
3. I am not sure if this was discussed in containerd meeting before. @dmcgowan @AkihiroSuda
4. Is this the only solution? I heard that there is another proposal that is based on blockfile snapshotter to do similar limit for disk and IO.