[question] Is possible to use Docker Server with TT enabled in kernel?

[ionutnechita]

Hi Hamad Al Marri,

Is it possible to have the TT in kernel and the docker server working properly? Is there a problem with CGROUPs?

// level=warning msg="Unable to find cpu cgroup in mounts"

A good day. Ionut.

[ptr1337]

@ionutnechita Should work, but i dont think you wont profit much on it. Since the scheduler is mostly thought to provide a low latency. But you ca try it.

[ionutnechita]

I tried today, but the docker containers don't start.

[ionutnechita]

Maybe tomorrow I'll give you more details.

[ptr1337]

@ionutnechita

Do you have autogroup disabled ?

[ionutnechita]

I use this options: CONFIG_TT_SCHED=y CONFIG_TT_ACCOUNTING_STATS=y

And AUTOGROUP is disabled.

Patch set: tt-5.15.patch - 0.3.7 high-hz.patch

[ptr1337]

@ionutnechita

Sorry, i was wrong. It does not work, since autogroup is disabled at default.

[hamadmarri]

hi @ionutnechita

could you please upload the .config

docker sometimes works without autogroup and sched_group

[ionutnechita]

I'm sending you 2 .config, one without TT and one with TT.

1. Without TT: config-5.15.9-lowlatency-non-tt.txt

2. With TT: config-5.15.9-lowlatency-with-tt.txt

My scenario:

1. Docker Server is installed on my host.
2. Use Kind tool for create new cluster. Docker Server is backend for new cluster.

./kind create cluster --config test.yml -v 1 Without TT, working correctly: Creating cluster "kind" ... DEBUG: docker/images.go:58] Image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 present locally ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 ✓ Preparing nodes 📦 📦 📦 📦 📦 📦 ✓ Configuring the external load balancer ⚖️ ✓ Writing configuration 📜 ✓ Starting control-plane 🕹️ ✓ Installing CNI 🔌 ✓ Installing StorageClass 💾 ✓ Joining more control-plane nodes 🎮 ✓ Joining worker nodes 🚜 Set kubectl context to "kind-kind" You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

./kind create cluster --config test.yml -v 1 With TT, not working correctly: Creating cluster "kind" ... DEBUG: docker/images.go:58] Image: kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6 present locally ✓ Ensuring node image (kindest/node:v1.21.1) 🖼 ✓ Preparing nodes 📦 📦 📦 📦 📦 📦 ✓ Configuring the external load balancer ⚖️ ✗ Writing configuration 📜 ERROR: failed to create cluster: failed to generate kubeadm config content: failed to get kubernetes version from node: failed to get file: command "docker exec --privileged kind-worker3 cat /kind/version" failed with error: exit status 1 Command Output: Error response from daemon: Container b411eec646c160287d2c770eefa0fffc3498d30ecb9debdd838b4966c309094f is not running Stack Trace: sigs.k8s.io/kind/pkg/errors.WithStack sigs.k8s.io/kind/pkg/errors/errors.go:59 sigs.k8s.io/kind/pkg/exec.(*LocalCmd).Run sigs.k8s.io/kind/pkg/exec/local.go:124 sigs.k8s.io/kind/pkg/cluster/internal/providers/docker.(*nodeCmd).Run sigs.k8s.io/kind/pkg/cluster/internal/providers/docker/node.go:146 sigs.k8s.io/kind/pkg/exec.OutputLines sigs.k8s.io/kind/pkg/exec/helpers.go:81 sigs.k8s.io/kind/pkg/cluster/nodeutils.KubeVersion sigs.k8s.io/kind/pkg/cluster/nodeutils/util.go:35 sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.getKubeadmConfig sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config/config.go:208 sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config.(*Action).Execute.func1.1 sigs.k8s.io/kind/pkg/cluster/internal/create/actions/config/config.go:90 sigs.k8s.io/kind/pkg/errors.UntilErrorConcurrent.func1 sigs.k8s.io/kind/pkg/errors/concurrent.go:30 runtime.goexit runtime/asm_amd64.s:1371

A good day. Ionut.

[ptr1337]

@ionutnechita

There was also some time ago a cfs release, maybe that is a try worth. Also you could use hamad's old scheduler, the cacule scheduler.

I also run that on my buildserver, the RDB version at it runs great.

Here you find the patches for 5.15 and 5.16.

https://github.com/CachyOS/cacule-cpu-scheduler

[hamadmarri]

Hi @ionutnechita

Could you please try with UCLAMP_TASK enabled and also with HZ=300 or 1000

[ionutnechita]

Hi @hamadmarri

I try with UCLAMP_TASK enabled and also with HZ=300. But not working.

config-5.15.9-lowlatency-with-tt-uclamp.txt

And i notice this in journalctl: journalctl -u docker ... level=warning msg="Unable to find cpu cgroup in mounts"

[ionutnechita]

I will test the next few days. Thank you for help.

Happy New Year.

[hamadmarri]

@ionutnechita

You might also try the BORE scheduler https://github.com/firelzrd/bore-scheduler

[ionutnechita]

Hi @hamadmarri

I tried with BORE scheduler and working fine. I think in the future I will use BORE or Cacule scheduler.

I don't think TT design supports that full docker support. But this TT scheduler is very powerful.

Thanks for support.

A good day.