containerd icon indicating copy to clipboard operation
containerd copied to clipboard

Containerd Image Pull Performance is Worse Than Docker

Open jeremyje opened this issue 4 years ago • 7 comments

Description The image pull pull performance on Windows is significantly worse for containerd compared to dockerd. There's a few issues at play here:

  1. Windows Defender CPU overhead.
  2. Seems like containerd is missing some pull optimizations that docker has.
  3. This is mostly observed in the extraction phase of the image pull.

Perf graphs coming soon.

Steps to reproduce the issue:

  1. crictl pull mcr.microsoft.com/windows/servercore:ltsc2019

Describe the results you received:

Describe the results you expected:

Output of containerd --version:

(paste your output here)

Any other relevant information:

jeremyje avatar Feb 02 '21 18:02 jeremyje

Tests on GCE:

Here are the results from the initial test I did manually on ltsc2019 based VM, with containerd 1.4.3 installed:

The unpacking was consuming most of the time (even when unpigz is enabled, and so the unpacking is offloaded)

ctr image pull mcr.microsoft.com/windows/servercore:1809 Start: December 15, 2020 12:00:24 AM Download completed: December 15, 2020 12:01:38 AM Unpacking (unpigz processes running): Start: December 15, 2020 12:01:38 AM Finished: December 15, 2020 12:14:10 AM Pull completed: December 15, 2020 12:17:48 AM

Noticed in the above case that unpigz is just running at ~0.5% cpu, containerd at ~20-30% cpu, and finally antimalware service (live scanning) at ~40-50% cpu. There was around ~13 mins out of 18 in the unpacking.

Tried one more run with the live scanning being disabled (Set-MpPreference -DisableRealtimeMonitoring $true):

The pure unpacking duration went down to ~5 mins (unpigz was up to 15% cpu or so) Unpacking (unpigz processes running): Start: December 15, 2020 1:06:51 AM Finished: December 15, 2020 1:11:48 AM The total here was ~10 minutes compared to ~18 mins above, as everything else took almost the same.

Further, I had an automated script that ran the same pulling-test, against various VM sizes with both containerd 1.4.1 and 1.4.3 (with Pigz on and off for every variation).

Here is a result summary for both ltsc2019 and 2004: 2019 sac2004

ibabou avatar Feb 02 '21 18:02 ibabou

Here is the script I used for installing containerd:

function Install-Containerd {
  $NODE_DIR = 'C:\node'
  $CNI_DIR = 'C:\node\cni'
  $CNI_CONFIG_DIR = 'C:\node\cni' # no config file is written

  if (-not (Test-Path $NODE_DIR)) {
    New-Item $NODE_DIR -ItemType 'directory' -Force | Out-Null
    New-Item $CNI_DIR -ItemType 'directory' -Force | Out-Null
  }
  
  $tmp_dir = 'C:\containerd_tmp'
  New-Item $tmp_dir -ItemType 'directory' -Force | Out-Null

  $version = $env:cd_ver
  $base_url = "https://github.com/containerd/containerd/releases/download"
  $tar_url = "$base_url/v$version/cri-containerd-cni-$version-windows-amd64.tar.gz"
  Invoke-WebRequest -Uri $tar_url -OutFile $tmp_dir\containerd.tar.gz -UseBasicParsing

  tar xzvf $tmp_dir\containerd.tar.gz -C $tmp_dir
  Move-Item -Force $tmp_dir\cni\*.exe $CNI_DIR\
  Move-Item -Force $tmp_dir\*.exe $NODE_DIR\
  Remove-Item -Force -Recurse $tmp_dir

  $config_dir = 'C:\containerd'
  New-Item $config_dir -ItemType 'directory' -Force | Out-Null
  Set-Content "$config_dir\config.toml" @"
[plugins.cri]
  sandbox_image = 'INFRA_CONTAINER_IMAGE'
[plugins.cri.cni]
  bin_dir = 'CNI_BIN_DIR'
  conf_dir = 'CNI_CONF_DIR'
"@.replace('INFRA_CONTAINER_IMAGE', 'e2eteam/pause:3.2').`
    replace('CNI_BIN_DIR', "$CNI_DIR").`
    replace('CNI_CONF_DIR', "$CNI_CONFIG_DIR")
}

function Start-Containerd {
  $log_dir = "C:\logs"
  New-Item $log_dir -ItemType 'directory' -Force | Out-Null
  C:\node\containerd.exe --register-service --config "C:\containerd\config.toml" --log-file "$log_dir\containerd.log"
  Start-Service containerd
  Write-Host "Started containerd"
}

ibabou avatar Feb 02 '21 18:02 ibabou

We are also observing that containerd image extraction sizes are larger than docker's.

jeremyje avatar Feb 17 '21 20:02 jeremyje

Related: https://github.com/containerd/containerd/issues/4937

awprice avatar Feb 25 '21 05:02 awprice

Closed by #6702

fuweid avatar Apr 06 '22 05:04 fuweid

@fuweid I think this issue was referring to just single pull performance is worse than docker which your change wouldn't have solved, I haven't gotten a chance to benchmark this yet but I'm going to reopen for now

dcantah avatar Apr 08 '22 19:04 dcantah

Related to #4819? While trying to figure out why kind load docker-image is so slow I found that docker save is surprisingly slow (14s to save a 1.8Gb image to SSD on my laptop) but ctr images import is slower still (31s).

jglick avatar Jan 18 '24 19:01 jglick