kondense icon indicating copy to clipboard operation
kondense copied to clipboard

Automated resources sizing tool for containers in kubernetes

Kondense

Go version Go Report Card GitHub release

drawing

Kondense is an automated resource sizing tool. It runs as a sidecar in kubernetes pods.

Background

Memory

Kondense uses memory pressure to apply just the right amount of memory on a container to page out the unused memory while not getting out-of-memory killed.How is memory calculated ?

CPU

Kondense resizes CPU based on CPU usage, default to 80%.How is CPU calculated ?

Requirements

On Kubernetes

  • The Kubernetes cluster must run on Linux.
  • Containerd version >= 1.6.9.
  • Kubernetes should have the feature gate InPlacePodVerticalScaling enabled.

On Containers

  • Containers should include the linux kernel version >= 4.20. Ensure the file /sys/fs/cgroup/memory.pressure exists in the container to verify it.

You can build your containers from the image alpine to be sure.

Example

Try the example on minikube:

minikube start --kubernetes-version=v1.29.2 --feature-gates=InPlacePodVerticalScaling=true
kubectl apply -f https://raw.githubusercontent.com/unagex/kondense/main/example/nginx.yaml

Let's say we have a pod running nginx that we want to Kondense:

apiVersion: v1
kind: Pod
metadata:
  name: kondense-test
spec:
  serviceAccountName: nginx-user
  containers:
  - name: nginx
    image: nginx:latest
    resources:
      limits:
        cpu: 100m
        memory: 100M

Add Kondense as a sidecar:

apiVersion: v1
kind: Pod
metadata:
  name: kondense-test
spec:
  serviceAccountName: nginx-user
  containers:
  - name: nginx
    image: nginx:latest
    resources:
      limits:
        cpu: 0.1
        memory: 100M
  - name: kondense
    image: kondense/kondense:1.1.0
    resources:
      limits:
        cpu: 80m
        memory: 50M

After adding the kondense container, the nginx container resources are updated without any container restart.

Notes:

  1. The pod should have a QoS of Guaranteed. In other words, we need to add resources limits for each containers.
  2. The service account nginx-user should have the following rules:
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch", "patch"]
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["create"]

Configuration

Kondense is configurable via environment variables in the kondense container.

Example

    ...
    - name: kondense
      image: kondense/kondense:1.1.0
      resources:
        limits:
          cpu: 80m
          memory: 50M
      env:
      - name: <CONTAINER NAME>_MEMORY_MIN
        value: "100m"

If we have a container named nginx in our pod, the variable name should be NGINX_MEMORY_MIN.

Environment variables

Global

Name Default value Description
EXCLUDE "" Comma separated list of containers to not kondense.

Memory

Name Default value Description
<CONTAINER NAME>_MEMORY_MIN 50M Minimum memory of the container. Kondense will never resize below that limit.
<CONTAINER NAME>_MEMORY_MAX 100G Maximum memory of the container. Kondense will never resize above that limit.
<CONTAINER NAME>_MEMORY_TARGET_PRESSURE 10000 Target memory pressure in microseconds. Kondense will take corrective actions to obtain it.
<CONTAINER NAME>_MEMORY_INTERVAL 10 Kondense targets cumulative memory delays over the sampling period of this interval in seconds.
<CONTAINER NAME>_MEMORY_MAX_INC 0.5 Maximum memory increase for one correction. e.g. 0.5 is a 50% increase.
<CONTAINER NAME>_MEMORY_MAX_DEC 0.02 Maximum memory decrease for one correction. e.g. 0.02 is a 2% decrease.
<CONTAINER NAME>_MEMORY_COEFF_INC 20 Coeff to increase memory when the memory pressure is bigger then the target memory pressure.
<CONTAINER NAME>_MEMORY_COEFF_DEC 10 Coeff to decrease memory when the memory pressure is smaller then the target memory pressure.

CPU

Name Default value Description
<CONTAINER NAME>_CPU_MIN 0.08 Minimum CPU of the container. Kondense will never resize below that limit.
<CONTAINER NAME>_CPU_MAX 100 Maximum CPU of the container. Kondense will never resize above that limit.
<CONTAINER NAME>_CPU_MAX_INC 0.5 Maximum CPU increase for one correction. e.g. 0.5 is a 50% increase.
<CONTAINER NAME>_CPU_MAX_DEC 0.1 Maximum CPU decrease for one correction. e.g. 0.1 is a 10% decrease.
<CONTAINER NAME>_CPU_TARGET_AVG 0.8 Target CPU average for the container. It is from 0 to 1. e.g. 0.8 means a target cpu usage of 80%.
<CONTAINER NAME>_CPU_INTERVAL 6 Target CPU average for the container. It is from 0 to 1. e.g. 0.8 means a target cpu usage of 80%.
<CONTAINER NAME>_CPU_COEFF 6 Used to calculate the new cpu limit when a cpu increase is needed. The higher the coeff, the higher the new cpu limit.

More