Performance for Postgres Container 2-4x Slower on Docker vs Native

[scottdfedorov]

Description

I am working on moving our codebase to Testcontainers and noticed the runtimes for our test suite are significantly longer when running with a Docker Postgres container versus the same code pointed to an installed Postgres natively. (The only change in the code is changing the jdbc URL to point to one or the other.)

I have created a repo with minimum required reproduction steps. https://github.com/scottdfedorov/tc-postgres-perf-demo

The benchmarks from this are showing that in fact between 2x and 4x difference between the two.

15:19:51.653 [main] INFO  org.testcontainers.repro.ReproExampleTest - STARTING RUN AGAINST DOCKER
15:21:41.815 [main] INFO  org.testcontainers.repro.ReproExampleTest - FINISHED RUN AGAINST DOCKER, TOOK 110161 MS
15:21:41.816 [main] INFO  org.testcontainers.repro.ReproExampleTest - STARTING RUN AGAINST NATIVE
15:22:26.748 [main] INFO  org.testcontainers.repro.ReproExampleTest - FINISHED RUN AGAINST NATIVE, TOOK 44932 MS

We have tested with multiple generations of M# Macs. I have also tried maxing out the resources in Docker Desktop and nearly every combination of Use Virtualization framework and the file sharing settings (VirtioFS, gRPC FUSE,...) The impact is the same.

Wondering what we can do to speed up the execution on ARM-based Mac's.

Reproduce

Checkout https://github.com/scottdfedorov/tc-postgres-perf-demo, run the script from root (or setup manually and run).

Expected behavior

No response

docker version

❯ docker version
Client:
 Version:           27.2.0
 API version:       1.47
 Go version:        go1.21.13
 Git commit:        3ab4256
 Built:             Tue Aug 27 14:14:45 2024
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.34.2 (167172)
 Engine:
  Version:          27.2.0
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.21.13
  Git commit:       3ab5c7d
  Built:            Tue Aug 27 14:15:41 2024
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.7.20
  GitCommit:        8fc6bcff51318944179630522a095cc9dbf9f353
 runc:
  Version:          1.1.13
  GitCommit:        v1.1.13-0-g58aa920
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

❯ docker info
Client:
 Version:    27.2.0
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2-desktop.1
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.2-desktop.2
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.34
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Alpha) (Docker Inc.)
    Version:  v0.0.15
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-desktop
  dev: Docker Dev Environments (Docker Inc.)
    Version:  v0.1.2
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-dev
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.25
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-extension
  feedback: Provide feedback, right in your terminal! (Docker Inc.)
    Version:  v1.0.5
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-feedback
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.3.0
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-init
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.13.0
    Path:     /Users/scottfedorov/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 6
 Server Version: 27.2.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  seccomp
   Profile: unconfined
  cgroupns
 Kernel Version: 6.10.4-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 7.752GiB
 Name: docker-desktop
 ID: 6b5225dc-c0c3-4297-a3af-8575a9d44276
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/scottfedorov/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: daemon is not using the default seccomp profile

Diagnostics ID

Diagnostics ID: 2E868F3A-FA50-47AE-882F-F642AB2CFC2A/20240926202545 (uploaded)

Additional Info

cc @eddumelendez

[scottdfedorov]

Also, discovered this issue before which is what made me initially play around with the virtualization settings, to no effect :( https://github.com/docker/for-mac/issues/5389 Sorry to @ you @stephen-turner, but you were so helpful there. Let me know if you have any ideas of what we can try or additional info I can supply.

[doringeman]

Hello @scottdfedorov, Thank you for the thorough description! We're currently working on a significant improvement in this area. We'll post here a build for testing soon, stay tuned!

[scottdfedorov]

Thanks @doringeman . What kinda of a timeline would you wager "soon" is? Weeks? Months?

Is there anything you might suggest we try to improve things in the meantime? I'm not sure where this bottleneck is, but the postgres container is supposedly not using a write to disk (fsync=off) so not sure its disk related. Any other points we might be able to speed it up on?

[doringeman]

I expect to have a build ready in the next couple of weeks.

[scottdfedorov]

Amazing, thanks! Let me know if I can help with anything.

[twk]

This sounds amazing! I'll be curious if your change fixes this issue from a few years ago. The linked issue has another benchmark that might be helpful for testing.

[scottdfedorov]

Hi @doringeman... Question for you: Is this new Docker VMM option supposed to be the improvement we've been waiting for?

I enabled that in my Docker desktop this morning and re-ran my init/test script (multiple times), and there's no improvement in the runtime difference between postgres native and postgres on docker.

08:02:49.546 [main] INFO  org.testcontainers.repro.ReproExampleTest - STARTING RUN AGAINST DOCKER
08:04:40.089 [main] INFO  org.testcontainers.repro.ReproExampleTest - FINISHED RUN AGAINST DOCKER, TOOK 110543 MS
08:04:40.090 [main] INFO  org.testcontainers.repro.ReproExampleTest - STARTING RUN AGAINST NATIVE
08:05:21.608 [main] INFO  org.testcontainers.repro.ReproExampleTest - FINISHED RUN AGAINST NATIVE, TOOK 41518 MS

[luetm]

@scottdfedorov, Did you find a solution? For us, this could mean a decision against Postgres if "inside-docker performance" is that bad.

[scottdfedorov]

@scottdfedorov, Did you find a solution? For us, this could mean a decision against Postgres if "inside-docker performance" is that bad.

Unfortunately not. I am still waiting for those supposed performance improvements. The last time I ran the benchmark I posted those results to no improvement. I haven't run them recently though. I might try to do it again later this week, but feel free to check out my repo if you wanna confirm yourself.