cri-o Failed to run cri-o 1.20 on Ubuntu 18.04

Failed to run cri-o 1.20 on Ubuntu 18.04

Open Drakkai opened this issue 3 years ago • 12 comments

Description

I tried to install kubernetes 1.20.2 using kubespray based on cri-o and Ubuntu 18.04. Unfortunately, the installation ended with an error starting crio.service. So I tried to install cri-o on a fresh system, but without success - I am not able to run cri-o version 1.20 on Ubuntu 18.04. I installed it according to the instructions on the website https://cri-o.io/. I get the same error for version 1.19.

root@test:~# systemctl start crio
Job for crio.service failed because the control process exited with error code.
See "systemctl status crio.service" and "journalctl -xe" for details.
root@test:~# journalctl -u crio
-- Logs begin at Sun 2021-02-14 15:00:46 CET, end at Sun 2021-02-14 15:02:52 CET. --
Feb 14 15:02:16 test systemd[1]: Starting Container Runtime Interface for OCI (CRI-O)...
Feb 14 15:02:16 test crio[2605]: time="2021-02-14T15:02:16+01:00" level=info msg="Starting CRI-O, version: 1.20.0, git: d388528dbed26b93c5bc1c89623607a1e597aa57(clean)"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.134115020+01:00" level=info msg="Node configuration value for hugetlb cgroup is true"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.134200522+01:00" level=info msg="Node configuration value for pid cgroup is true"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.134305024+01:00" level=error msg="Node configuration validation for memoryswap cgroup failed: node not configured with memory swap"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.134459526+01:00" level=info msg="Node configuration value for memoryswap cgroup is false"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.135773751+01:00" level=info msg="Node configuration value for systemd CollectMode is true"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.187879808+01:00" level=info msg="Overlay test mount did not indicate whether or not metacopy is being used: failed to mount overlay for metacopy check with \"nodev,metacopy=on\" options: invalid argument"
Feb 14 15:02:16 test crio[2605]: time="2021-02-14 15:02:16.188078612+01:00" level=fatal msg="Validating root config: failed to get store to set defaults: failed to mount overlay for metacopy check with \"nodev,metacopy=on\" options: invalid argument"
Feb 14 15:02:16 test systemd[1]: crio.service: Main process exited, code=exited, status=1/FAILURE
Feb 14 15:02:16 test systemd[1]: crio.service: Failed with result 'exit-code'.
Feb 14 15:02:16 test systemd[1]: Failed to start Container Runtime Interface for OCI (CRI-O).

Steps to reproduce the issue:

Install Ubuntu 18.04
Add repositories and keys
Install cri-o and cri-o-runc
Reload systemd
Failed to start crio.service

Describe the results you received: Validating root config: failed to get store to set defaults: failed to mount overlay for metacopy check with \"nodev,metacopy=on\" options: invalid argument

Describe the results you expected: Run crio.service without problem.

Additional information you deem important (e.g. issue happens only occasionally):

Output of crio --version:

INFO[0000] Starting CRI-O, version: 1.20.0, git: d388528dbed26b93c5bc1c89623607a1e597aa57(clean)
crio version 1.20.0
Version:       1.20.0
GitCommit:     d388528dbed26b93c5bc1c89623607a1e597aa57
GitTreeState:  clean
BuildDate:     2020-12-16T11:12:35Z
GoVersion:     go1.15.2
Compiler:      gc
Platform:      linux/amd64
Linkmode:      dynamic

Additional environment details (AWS, VirtualBox, physical, etc.): Ubuntu 18.04 on Hyper-V

root@test:~# uname -a
Linux test 4.15.0-135-generic #139-Ubuntu SMP Mon Jan 18 17:38:24 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
root@test:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.5 LTS
Release:        18.04
Codename:       bionic

Feb 14 '21 14:02 Drakkai

can you try to run sed -i 's/,metacopy=on//g' /etc/containers/storage.conf and see if that works? It seems ubuntu 18.04 doesn't support the option

Feb 15 '21 14:02 haircommander

Thanks, it works. Maybe it should be mentioned in the documentation?

root@test:~# sed -i 's/,metacopy=on//g' /etc/containers/storage.conf
root@test:~# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: https://github.com/cri-o/cri-o
root@test:~# systemctl start crio
root@test:~# systemctl status crio
● crio.service - Container Runtime Interface for OCI (CRI-O)
   Loaded: loaded (/usr/lib/systemd/system/crio.service; disabled; vendor preset: enabled)
   Active: active (running) since Mon 2021-02-15 18:19:21 CET; 2s ago
     Docs: https://github.com/cri-o/cri-o
 Main PID: 2705 (crio)
    Tasks: 8
   CGroup: /system.slice/crio.service
           └─2705 /usr/bin/crio

Feb 15 '21 17:02 Drakkai

we should really fix this in the package itself, it isn't too hard to auto-detect, and not worth end-user time/energy :)

Feb 15 '21 17:02 haircommander

thanks for the report @Drakkai

Feb 15 '21 17:02 haircommander

An easy rule would be kernel >= 4.19 or kernel = 4.18.0 (EL 8.x)

Feb 15 '21 19:02 champtar

One of the painful part is that ubuntu (18.04) has different kernel versions in the cloud than in normal images. Cleanest way to fix this is to use today failing check and turn that into an autodetection rule, so this also handle custom kernel

Feb 15 '21 19:02 champtar

we should really fix this in the package itself, it isn't too hard to auto-detect, and not worth end-user time/energy :)

on the other hand, you can of course keep it for some nameless poor soul to learn about how to troubleshoot low level storage issues between CRI and Kubernetes API :)