containerd
containerd copied to clipboard
Containerd Image Pull Performance is Worse Than Docker
Description The image pull pull performance on Windows is significantly worse for containerd compared to dockerd. There's a few issues at play here:
- Windows Defender CPU overhead.
- Seems like containerd is missing some pull optimizations that docker has.
- This is mostly observed in the extraction phase of the image pull.
Perf graphs coming soon.
Steps to reproduce the issue:
- crictl pull mcr.microsoft.com/windows/servercore:ltsc2019
Describe the results you received:
Describe the results you expected:
Output of containerd --version
:
(paste your output here)
Any other relevant information:
Tests on GCE:
Here are the results from the initial test I did manually on ltsc2019 based VM, with containerd 1.4.3 installed:
The unpacking was consuming most of the time (even when unpigz is enabled, and so the unpacking is offloaded)
ctr image pull mcr.microsoft.com/windows/servercore:1809 Start: December 15, 2020 12:00:24 AM Download completed: December 15, 2020 12:01:38 AM Unpacking (unpigz processes running): Start: December 15, 2020 12:01:38 AM Finished: December 15, 2020 12:14:10 AM Pull completed: December 15, 2020 12:17:48 AM
Noticed in the above case that unpigz is just running at ~0.5% cpu, containerd at ~20-30% cpu, and finally antimalware service (live scanning) at ~40-50% cpu. There was around ~13 mins out of 18 in the unpacking.
Tried one more run with the live scanning being disabled (Set-MpPreference -DisableRealtimeMonitoring $true):
The pure unpacking duration went down to ~5 mins (unpigz was up to 15% cpu or so) Unpacking (unpigz processes running): Start: December 15, 2020 1:06:51 AM Finished: December 15, 2020 1:11:48 AM The total here was ~10 minutes compared to ~18 mins above, as everything else took almost the same.
Further, I had an automated script that ran the same pulling-test, against various VM sizes with both containerd 1.4.1 and 1.4.3 (with Pigz on and off for every variation).
Here is a result summary for both ltsc2019 and 2004: 2019 sac2004
Here is the script I used for installing containerd:
function Install-Containerd {
$NODE_DIR = 'C:\node'
$CNI_DIR = 'C:\node\cni'
$CNI_CONFIG_DIR = 'C:\node\cni' # no config file is written
if (-not (Test-Path $NODE_DIR)) {
New-Item $NODE_DIR -ItemType 'directory' -Force | Out-Null
New-Item $CNI_DIR -ItemType 'directory' -Force | Out-Null
}
$tmp_dir = 'C:\containerd_tmp'
New-Item $tmp_dir -ItemType 'directory' -Force | Out-Null
$version = $env:cd_ver
$base_url = "https://github.com/containerd/containerd/releases/download"
$tar_url = "$base_url/v$version/cri-containerd-cni-$version-windows-amd64.tar.gz"
Invoke-WebRequest -Uri $tar_url -OutFile $tmp_dir\containerd.tar.gz -UseBasicParsing
tar xzvf $tmp_dir\containerd.tar.gz -C $tmp_dir
Move-Item -Force $tmp_dir\cni\*.exe $CNI_DIR\
Move-Item -Force $tmp_dir\*.exe $NODE_DIR\
Remove-Item -Force -Recurse $tmp_dir
$config_dir = 'C:\containerd'
New-Item $config_dir -ItemType 'directory' -Force | Out-Null
Set-Content "$config_dir\config.toml" @"
[plugins.cri]
sandbox_image = 'INFRA_CONTAINER_IMAGE'
[plugins.cri.cni]
bin_dir = 'CNI_BIN_DIR'
conf_dir = 'CNI_CONF_DIR'
"@.replace('INFRA_CONTAINER_IMAGE', 'e2eteam/pause:3.2').`
replace('CNI_BIN_DIR', "$CNI_DIR").`
replace('CNI_CONF_DIR', "$CNI_CONFIG_DIR")
}
function Start-Containerd {
$log_dir = "C:\logs"
New-Item $log_dir -ItemType 'directory' -Force | Out-Null
C:\node\containerd.exe --register-service --config "C:\containerd\config.toml" --log-file "$log_dir\containerd.log"
Start-Service containerd
Write-Host "Started containerd"
}
We are also observing that containerd image extraction sizes are larger than docker's.
Related: https://github.com/containerd/containerd/issues/4937
Closed by #6702
@fuweid I think this issue was referring to just single pull performance is worse than docker which your change wouldn't have solved, I haven't gotten a chance to benchmark this yet but I'm going to reopen for now
Related to #4819? While trying to figure out why kind load docker-image
is so slow I found that docker save
is surprisingly slow (14s to save a 1.8Gb image to SSD on my laptop) but ctr images import
is slower still (31s).