Containers take exponentially longer time to start

Open f1amy opened this issue 1 year ago • 1 comments

Description

Hello, I run a project with ~12 sets of Docker Compose applications, all connected by networks (databases, cache, webservers and php workers and applications), all have volumes and volume mounts for code or databases. I launch my project via docker compose with makefiles, firstly launching application services, then sequentially each application one by one.

Recently, I did an update to Ubuntu 24.04 LTS, update WSL (wsl --update) on Windows 11. After the update, I could see that cold-starting the project takes extremely long time (20-30 minutes) compared to 1-3 minutes prior. The behavior is as follows: first containers from application services start pretty fast, then with each application set it takes them 20-30 seconds more to start, ending with over 170 seconds to start. The CPU usage seems to be the issue, but I could not find high cpu utilization processes with top when this happens.

This makes developing the project impossible, because containers would crash and restart before its required dependencies could be installed for setup (this used to work perfectly) / up container, exec into it, install dependencies/warm cache.

Reproduce

clone https://github.com/f1amy/containers-slow-start
make init
make build
make setup

Expected behavior

Containers should start fast enough (to hit application error).

docker version

Client:
 Version:           27.1.1
 API version:       1.46
 Go version:        go1.21.12
 Git commit:        6312585
 Built:             Tue Jul 23 19:55:52 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Desktop  ()
 Engine:
  Version:          27.1.1
  API version:      1.46 (minimum version 1.24)
  Go version:       go1.21.12
  Git commit:       cc13f95
  Built:            Tue Jul 23 19:57:19 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.19
  GitCommit:        2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc:
  Version:          1.7.19
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    27.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1-desktop.1
    Path:     /usr/local/lib/docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /usr/local/lib/docker/cli-plugins/docker-debug
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /usr/local/lib/docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /usr/local/lib/docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /usr/local/lib/docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.11.0
    Path:     /usr/local/lib/docker/cli-plugins/docker-scout

Server:
 Containers: 61
  Running: 25
  Paused: 0
  Stopped: 36
 Images: 63
 Server Version: 27.1.1
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
 Kernel Version: 5.15.153.1-microsoft-standard-WSL2
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 7.448GiB
 Name: docker-desktop
 ID: 924e3661-0343-4b40-8d59-df70b8ad83f6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///var/run/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support
WARNING: daemon is not using the default seccomp profile

Diagnostics ID

5F033B04-356E-4708-9131-A4943D70ED24/20240908045633

Additional Info

Could not get to generate diagnostics ID from target machine (v4.33.1), but the issue seems to be reproducible on other with similar environment.

Sep 08 '24 05:09 f1amy

It seems I spent entire evening, night, and morning debugging this issue only to realize that I had enabled low power mode on my laptop 🤪

The issue generally seems to exist, just without cpu throttling it is much more bearable so I haven't paid attention earlier.

At this point I suggest anyone who is deploying applications the same way, rewrite setup script in a way that does not require up of constantly restarting containers (as they don't have dependencies at cold start).

There could could have been a change in docker engine in how it (less) tolerates restarting containers in up/down commands, so this is the case someone could take a look if that was desirable.

Sep 08 '24 09:09 f1amy