runtime
runtime copied to clipboard
Published
20 hours ago •
clearcontainers
clearcontainers
Start qemu failed in CC 3.0.0 (works fine in CC 2.2.4)
Below error logs indicated there have some issue in qemu command line. Interesting my machine can ran CC 2.2.4 successfully.
Is there anyone can give me some tips? Thanks.
$sudo cat /var/lib/clear-containers/runtime/runtime.log
2017-09-27 15:34:39.388233786 +0800 CST:22918:cc-runtime:info:cc-runtime (version 3.0.0, commit 7fc11d5) called as: [create --bundle /var/run/docker/libcontainerd/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e --console-socket /tmp/pty454214293/pty.sock --pid-file /run/docker/libcontainerd/containerd/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/init/pid 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e]
2017-09-27 15:34:39.414433824 +0800 CST:22918:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-27 15:34:40.933751823 +0800 CST:22918:cc-runtime:info:No sockets from configuration
2017-09-27 15:34:40.956700885 +0800 CST:22918:cc-runtime:info:Device details for container 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e: Major:253, Minor:1, MountPoint:/var/lib/docker/devicemapper/mnt/7d7c67d334d72e9bf34b2792c68e692cb966afca78c88a0ec99dea2a4794d6b5
2017-09-27 15:34:40.982031478 +0800 CST:22918:cc-runtime:info:Block Device path /dev/mapper/docker-8:2-2402074-7d7c67d334d72e9bf34b2792c68e692cb966afca78c88a0ec99dea2a4794d6b5 detected for container with fstype : 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e
%!(EXTRA string=xfs)
2017-09-27 15:34:41.310825787 +0800 CST:22918:cc-runtime:info:launching qemu with: [-name pod-7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e -uuid 37643065-6562-3030-3234-636330323236 -machine pc,accel=kvm,kernel_irqchip,nvdimm -cpu host -qmp unix:/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/monitor.sock,server,nowait -qmp unix:/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/ctrl.sock,server,nowait -m 2048M,slots=2,maxmem=97517M -smp 24,cores=24,threads=1,sockets=1 -device virtio-9p-pci,fsdev=ctr-9p-0,mount_tag=ctr-rootfs-0 -fsdev local,id=ctr-9p-0,path=/var/lib/docker/devicemapper/mnt/7d7c67d334d72e9bf34b2792c68e692cb966afca78c88a0ec99dea2a4794d6b5/rootfs,security_model=none -device virtio-serial-pci,id=serial0 -device virtconsole,chardev=charconsole0,id=console0 -chardev socket,id=charconsole0,path=/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/console.sock,server,nowait -device nvdimm,id=nv0,memdev=mem0 -object memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-containers.img,size=235929600 -device virtserialport,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch0,path=/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/hyper.sock,server,nowait -device virtserialport,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -chardev socket,id=charch1,path=/run/virtcontainers/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e/tty.sock,server,nowait -device virtio-9p-pci,fsdev=extra-9p-hyperShared,mount_tag=hyperShared -fsdev local,id=extra-9p-hyperShared,path=/tmp/hyper/shared/pods/7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e,security_model=none -device virtio-blk,drive=drive-7d0eeb0024cc02261c1592260,scsi=off,config-wce=off -drive id=drive-7d0eeb0024cc02261c1592260,file=/dev/mapper/docker-8:2-2402074-7d7c67d334d72e9bf34b2792c68e692cb966afca78c88a0ec99dea2a4794d6b5,aio=threads,format=raw,if=none -device driver=virtio-net-pci,netdev=network-0,mac=02:42:ac:11:00:02 -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=discard -vga none -no-user-config -nodefaults -nographic -daemonize -kernel /usr/share/clear-containers/vmlinuz.container -append root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e::off::]
2017-09-27 15:34:41.350440609 +0800 CST:22918:cc-runtime:error:Unable to launch qemu: exit status 1
2017-09-27 15:34:41.374005133 +0800 CST:22918:cc-runtime:error:qemu-system-x86_64: -netdev tap,id=network-0,ifname=tap0,downscript=no,script=no,vhost=on: tap: open vhost char device failed: No such file or directory
2017-09-27 15:34:42.310571277 +0800 CST:22918:cc-runtime:error:Did not receive the pod started notification
2017-09-27 15:34:42.357430479 +0800 CST:23032:cc-runtime:info:cc-runtime (version 3.0.0, commit 7fc11d5) called as: [delete 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e]
2017-09-27 15:34:42.374332063 +0800 CST:23032:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-27 15:34:43.911450546 +0800 CST:23032:cc-runtime:info:Shim(s) stopped
2017-09-27 15:34:43.943192904 +0800 CST:23032:cc-runtime:error:dial unix /var/run/clear-containers/proxy.sock: connect: connection refused
2017-09-27 15:34:44.010752112 +0800 CST:23487:cc-runtime:info:cc-runtime (version 3.0.0, commit 7fc11d5) called as: [delete 7d0eeb0024cc02261c15922600c79e342d59a860593dd07fc955e126bc9a6d4e]
2017-09-27 15:34:44.044182951 +0800 CST:23487:cc-runtime:info:Using configuration file "/etc/clear-containers/configuration.toml"
2017-09-27 15:34:45.612318276 +0800 CST:23487:cc-runtime:error:Can not move from stopped to stopped
Is this the same system that cc-check fails on (your #634 issue)?
Note that that log shows you are using /etc/clear-containers/configuration.toml. Have you seen:
- https://lists.01.org/pipermail/cc-devel/2017-September/000122.html
Also, it would be extremely useful if you could fill out the template you should have seen. Specifically, if you can run the cc-collect-data.sh script, that would be helpful. I notice that you are using the latest OBS release. That script isn't available in those package versions, but you can either copy the built file and run on your system (or just paste the output of sudo cc-runtime cc-env to this issue).
@jodh-intel it is the same machine with the cc-check failed.
I've changed the configuration file to /usr/share/defaults/clear-containers/configuration.toml
$sudo cc-runtime cc-env
[Meta]
Version = "1.0.3"
[Runtime]
[Runtime.Version]
Semver = "3.0.0"
Commit = "7fc11d5"
OCI = "1.0.0-rc5"
[Runtime.Config]
GlobalLogPath = "/var/lib/clear-containers/runtime/runtime.log"
[Runtime.Config.Location]
Path = "/usr/share/defaults/clear-containers/configuration.toml"
Resolved = "/usr/share/defaults/clear-containers/configuration.toml"
[Hypervisor]
MachineType = "pc"
Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
[Hypervisor.Location]
Path = "/usr/bin/qemu-system-x86_64"
Resolved = "/usr/bin/qemu-system-x86_64"
[Image]
Path = "/usr/share/clear-containers/clear-containers.img"
Resolved = "/usr/share/clear-containers/clear-17270-containers.img"
[Kernel]
Parameters = ""
[Kernel.Location]
Path = "/usr/share/clear-containers/vmlinuz.container"
Resolved = "/usr/share/clear-containers/vmlinuz-4.9.47-77.2.container"
[Proxy]
Type = "ccProxy"
Version = "Version: 3.0.0+git.c66f21e"
URL = "unix:///var/run/clear-containers/proxy.sock"
[Shim]
Type = "ccShim"
Version = "shim version: 3.0.0-alpha.4 (commit: 739e5e2)"
[Shim.Location]
Path = "/usr/libexec/clear-containers/cc-shim"
Resolved = "/usr/libexec/clear-containers/cc-shim"
[Agent]
Type = "hyperstart"
Version = "<<unknown>>"
[Agent.PauseBin]
Path = "/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"
Resolved = "/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"
[Host]
Kernel = "3.10.0-327.ali2011.alios7.x86_64"
CCCapable = false
[Host.Distro]
Name = "Alibaba Group Enterprise Linux Server"
Version = "7.2"
[Host.CPU]
Vendor = "GenuineIntel"
Model = "Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz"
@jovizhangwei i got the same error here is my configuration.toml file
#cat /etc/clear-containers/configuration.toml
# XXX: Warning: this file is auto-generated from file "".
[hypervisor.qemu]
path = "/usr/local/bin/qemu-lite-system-x86_64"
kernel = "/usr/local/share/clear-containers/vmlinuz.container"
image = "/usr/local/share/clear-containers/clear-containers.img"
machine_type = "pc"
# Optional space-separated list of options to pass to the guest kernel.
# For example, use `kernel_params = "vsyscall=emulate"` if you are having
# trouble running pre-2.15 glibc
kernel_params = ""
# Default number of vCPUs per POD/VM:
# unspecified or 0 --> will be set to 1
# < 0 --> will be set to the actual number of physical cores
# > 0 <= 255 --> will be set to the specified number
# > 255 --> will be set to 255
default_vcpus = -1
# Default memory size in MiB for POD/VM.
# If unspecified then it will be set 2048 MiB.
#default_memory = 2048
disable_block_device_use = false
# Enable pre allocation of VM RAM, default false
# Enabling this will result in lower container density
# as all of the memory will be allocated and locked
# This is useful when you want to reserve all the memory
# upfront or in the cases where you want memory latencies
# to be very predictable
# Default false
#enable_mem_prealloc = true
# Enable swap of vm memory. Default false.
# The behaviour is undefined if mem_prealloc is also set to true
#enable_swap = true
# Debug changes the default hypervisor and kernel parameters to
# enable debug output where available.
# Default false
# these logs can be obtained in the cc-proxy logs when the
# proxy is set to run in debug mode
# /usr/libexec/clear-containers/cc-proxy -log debug
# or by stopping the cc-proxy service and running the cc-proxy
# explicitly using the same command line
#
enable_debug = true
[proxy.cc]
url = "unix:///var/run/clear-containers/proxy.sock"
[shim.cc]
path = "/usr/local/libexec/clear-containers/cc-shim"
[agent.hyperstart]
pause_root_path = "/usr/local/var/lib/clear-containers/runtime/bundles/pause_bundle"
## Uncomment to enable the global logging to the default path.
#[runtime]
global_log_path = "/usr/local/var/lib/clear-containers/runtime/runtime.log"
cc-runtime cc-check output
#cc-runtime cc-check
INFO[0000] Found CPU attribute "Intel Architecture CPU" (GenuineIntel)
INFO[0000] Found CPU flag "Virtualization support" (vmx)
INFO[0000] Found CPU flag "64Bit CPU" (lm)
INFO[0000] Found CPU flag "SSE4.1" (sse4_1)
INFO[0000] Found kernel module "Kernel-based Virtual Machine" (kvm)
INFO[0000] Found kernel module "Intel KVM" (kvm_intel)
ERRO[0000] ERROR: kernel module "Intel KVM" parameter "nested" has value "N" (expected "Y")
ERROR: kernel module "Intel KVM" parameter "nested" has value "N" (expected "Y")
here is my cc-runtime cc-env output
#cc-runtime --cc-config=/etc/clear-containers/configuration.toml cc-env
[Meta]
Version = "1.0.4"
[Runtime]
[Runtime.Version]
Semver = "3.0.0-beta.2"
Commit = "d9897e1fa8579d456799c11d3a82e0acc6e7f1ec"
OCI = "1.0.0-rc5"
[Runtime.Config]
Path = "/etc/clear-containers/configuration.toml"
GlobalLogPath = ""
[Hypervisor]
MachineType = "pc"
Version = "QEMU emulator version 2.7.0, Copyright (c) 2003-2016 Fabrice Bellard and the QEMU Project developers"
Path = "/opt/vlinux/2.2.4/qemu-lite/2.7.0-e84eba1324/bin/qemu-system-x86_64"
[Image]
Path = "/usr/local/share/clear-containers/clear-containers.img"
[Kernel]
Path = "/usr/local/share/clear-containers/vmlinuz.container"
Parameters = ""
[Proxy]
Type = "ccProxy"
Version = "<<unknown>>"
URL = "unix:///var/run/clear-containers/proxy.sock"
[Shim]
Type = "ccShim"
Version = "shim version: 3.0.0-alpha.4 (commit: 4f2b7daf521934bdefc2748be235cfb7bc265736)"
Path = "/usr/local/libexec/clear-containers/cc-shim"
[Agent]
Type = "hyperstart"
Version = "<<unknown>>"
PauseBinPath = "/usr/local/var/lib/clear-containers/runtime/bundles/pause_bundle/bin/pause"
[Host]
Kernel = "3.10.0-327.ali2012_cgroup_pids_test_713.alios7.x86_64"
CCCapable = false
[Host.Distro]
Name = "Alibaba Group Enterprise Linux Server"
Version = "7.2"
[Host.CPU]
Vendor = "GenuineIntel"
Model = "Intel(R) Xeon(R) CPU X5670 @ 2.93GHz"
Hi @jovizhangwei - the problem is likely to be that your kernel doesn't have vhost capabilities. However, due to the way cc-check currently works (and because you have nesting disabled), you don't get to see that error from cc-check. I've raised #638 so we can improve cc-check.
Hi @jovizhangwei - can you try running the following and then seeing if you can start a container?
$ for m in vhost vhost_net; do sudo modprobe $m; done
@jodh-intel
$for m in vhost vhost_net; do sudo modprobe $m; done
modprobe: FATAL: Module vhost not found.
modprobe: FATAL: Module vhost_net not found.
Why I can ran CC 2.2.4? without vhost.
@jovizhangwei With 2.2.4 we were not enabling vhost by default. With 3.0 we assume vhost is available.
I just opened a virtcontainers issue for that: https://github.com/containers/virtcontainers/issues/385 and will get a fix for that in the next few days.
@jovizhangwei FWIW, vhost-net provides significant networking performance improvements with KVM. Is there any reason why it is not enabled with your kernel ?
@sameo After upgraded my kernel, vhost and vhost_net modules was loaded, then I found another error, please take a look at https://github.com/clearcontainers/runtime/issues/642
Thanks