dagger icon indicating copy to clipboard operation
dagger copied to clipboard
verified publisher icon dagger

🐞 `dagger functions` can be up to `12.5x` slower on certain hosts

Open gerhard opened this issue 1 year ago • 0 comments

What is the issue?

This is a follow-up to:

  • https://github.com/dagger/dagger/issues/8395

Given the following host:

noefetch
             /////////////                gerhard@h22
         /////////////////////            -----------
      ///////*767////////////////         OS: Pop!_OS 22.04 LTS x86_64
    //////7676767676*//////////////       Kernel: 6.9.3-76060903-generic
   /////76767//7676767//////////////      Uptime: 31 days, 7 hours, 31 mins
  /////767676///*76767///////////////     Packages: 2346 (dpkg), 13 (brew), 26 (flatpak)
 ///////767676///76767.///7676*///////    Shell: bash 5.1.16
/////////767676//76767///767676////////   Resolution: 1024x768
//////////76767676767////76767/////////   Terminal: /dev/pts/2
///////////76767676//////7676//////////   CPU: AMD Ryzen 7 5800X (16) @ 3.800GHz
////////////,7676,///////767///////////   GPU: NVIDIA 3f:00.0 NVIDIA Corporation Device 2704
/////////////*7676///////76////////////   Memory: 15210MiB / 128719MiB
///////////////7676////////////////////
 ///////////////7676///767////////////
  //////////////////////'////////////
   //////.7676767676767676767,//////
    /////767676767676767676767/////
      ///////////////////////////
         /////////////////////
             /////////////
docker info
Client: Docker Engine - Community
 Version:    27.1.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 11
 Server Version: 27.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2bf793ef6dc9a18e00cb12efb64355c2c9d5eb41
 runc version: v1.1.13-0-g58aa920
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.9.3-76060903-generic
 Operating System: Pop!_OS 22.04 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 125.7GiB
 Name: h22
 ID: 1c93b978-5459-4719-a1aa-92422d716091
 Docker Root Dir: /data-2024-04-10/docker
 Debug Mode: false
 Username: infradagger
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

dagger init --sdk=go --name=foo takes 17s in an empty directory https://dagger.cloud/dagger/traces/16339e06ad4f1364f6c5ae164a7cb18b

dagger functions takes 4s the first time https://dagger.cloud/dagger/traces/621b9e3901b585cd260f62a9c79c949e

dagger functions takes 1.5s the second time https://dagger.cloud/dagger/traces/83f2d9a9459f9b62d22397ed384bd031

So far, so good.

Same host, running the same command inside a subdir of a fresh clone of the dagger/dagger repository:

cd $(mktemp -d)
git clone https://github.com/dagger/dagger
cd dagger
mkdir foo
cd foo

dagger init --sdk=go --name=foo takes 25s in this dagger/dagger/foo directory https://dagger.cloud/dagger/traces/65ff21e193c7b7681d862aaf6253b047

dagger functions takes 25s the first time https://dagger.cloud/dagger/traces/dbf4420a92e4874e38fd8fd67b73c96f

dagger functions takes 22s the second time https://dagger.cloud/dagger/traces/7f2f5efad11460ee07d0728a2cbf460c

This is completely unexpected. All that time is spent in moduleSource(refString:"./foo", relHostPath:".") image

What's even more peculiar is that this behaviour is inconsistent between different hosts.

Given the following host:

noefetch
            .-/+oossssoo+/-.               gerhard@h24
        `:+ssssssssssssssssss+:`           -----------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 24.04 LTS x86_64
    .ossssssssssssssssssdMMMNysssso.       Host: Intel Z790 Ddr5
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Kernel: 6.8.0-40-generic
  +ssssssssshmydMMMMMMMNddddyssssssss+     Uptime: 7 hours, 7 mins
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Packages: 1724 (dpkg), 18 (brew), 13 (snap)
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Shell: bash 5.2.21
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Resolution: 2560x1440
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   Terminal: /dev/pts/0
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: 13th Gen Intel i9-13900KS (32) @ 3.200GHz
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   GPU: NVIDIA GeForce RTX 4090
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Memory: 2250MiB / 192932MiB
 /sssssssshNMMMyhhyyyyhdNMMMNhssssssss/
  +sssssssssdmydMMMMMMMMddddyssssssss+
   /ssssssssssshdmNNNNmyNMMMMhssssss/
    .ossssssssssssssssssdMMMNysssso.
      -+sssssssssssssssssyyyssss+-
        `:+ssssssssssssssssss+:`
            .-/+oossssoo+/-.
docker info
Client: Docker Engine - Community
 Version:    27.1.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.16.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 4
Server Version: 27.1.2
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8fc6bcff51318944179630522a095cc9dbf9f353
runc version: v1.1.13-0-g58aa920
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.8.0-40-generic
Operating System: Ubuntu 24.04 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 188.4GiB
Name: h24
ID: 2637752e-9a21-47b5-adb0-844c1c544e90
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

cd $(mktemp -d)
git clone https://github.com/dagger/dagger
cd dagger
mkdir foo
cd foo

dagger init --sdk=go --name=foo takes 15s in this dagger/dagger/foo directory https://dagger.cloud/dagger/traces/e2d70104b8fb18faddc976d9d18229bc

dagger functions takes 4s the first time https://dagger.cloud/dagger/traces/9a53db92c4f4e86ad633951b450ae64e

dagger functions takes 2s the second time https://dagger.cloud/dagger/traces/29e336eb13ae7db3c55f9a6a9a144709

How can this second host be 12x slower than the first host? There is no disk or CPU contention. Both hosts are running on the same network, with access to the same WAN. Here is a side-by-side comparison (left is the slow host, right is the fast one):

https://github.com/user-attachments/assets/0dfbf8af-afe9-43fe-aa97-d47bbf5f5dd3

Dagger version

dagger v0.12.7 (registry.dagger.io/engine:v0.12.7) linux/amd64

Steps to reproduce

cd $(mktemp -d)
git clone https://github.com/dagger/dagger
cd dagger
mkdir foo
cd foo

Log output

No response

gerhard avatar Sep 10 '24 18:09 gerhard