cc-oci-runtime
cc-oci-runtime copied to clipboard
Unable to run Clear Conntainers on Ubuntu 16.04 Server
Unable to run Clear Containers on Ubuntu 16.04 Server using docker as it gets timed out waiting for container.
As compatibility for Docker 17.06 is not yet supported as given here, I have setup Docker 17.03.2.
I started a docker daemon using this.
sudo dockerd --add-runtime cor=/usr/bin/cc-oci-runtime.sh --default-runtime=cor --storage-driver=vfs
Output of sudo docker info
is as follows.
Containers: 4
Running: 0
Paused: 0
Stopped: 4
Images: 1
Server Version: 17.03.2-ce
Storage Driver: vfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: cor runc
Default Runtime: cor
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.12.7
Operating System: Ubuntu 16.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 23.53 GiB
Name: SDN-NFV-Server
ID: 3VQW:Q5C5:5OGL:WVHZ:NSQY:6M3H:LJ6Z:Z5L4:YER6:62IT:3YYY:J6CP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Now, when I do sudo docker run -ti --runtime=runc hello-world
, everything works fine.
But, when I do sudo docker run -ti hello-world
, the output after two minutes is the following.
docker: Error response from daemon: containerd: container did not start before the specified timeout.
The error on the docker daemon is the following.
ERRO[0186] containerd: start container error=containerd: container did not start before the specified timeout id=d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
ERRO[0186] Create container failed with error: containerd: container did not start before the specified timeout
ERRO[0186] Handler for POST /v1.27/containers/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/start returned error: containerd: container did not start before the specified timeout
Output of journalctl -u cc-proxy.service
is this.
-- Logs begin at Fri 2017-08-18 21:54:28 IST, end at Fri 2017-08-18 22:09:34 IST. --
Aug 18 22:00:57 SDN-NFV-Server systemd[1]: Stopped Clear Containers Proxy.
Aug 18 22:00:57 SDN-NFV-Server systemd[1]: Started Clear Containers Proxy.
Aug 18 22:00:57 SDN-NFV-Server cc-proxy[1831]: I0818 22:00:57.467281 1831 proxy.go:381] proxy started
Aug 18 22:04:00 SDN-NFV-Server cc-proxy[1831]: I0818 22:04:00.988991 1831 proxy.go:71] [client #1] client connected
Aug 18 22:04:01 SDN-NFV-Server cc-proxy[1831]: I0818 22:04:01.298320 1831 proxy.go:81] [client #1] hello(containerId=d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b,ctlSerial=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-ctl.sock,ioSerial=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-tty.sock,console=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/console.sock)
Aug 18 22:06:00 SDN-NFV-Server cc-proxy[1831]: I0818 22:06:00.877399 1831 proxy.go:71] [client #2] client connected
Aug 18 22:06:00 SDN-NFV-Server cc-proxy[1831]: I0818 22:06:00.877635 1831 proxy.go:81] [client #2] attach(containerId=d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b)
Aug 18 22:06:00 SDN-NFV-Server cc-proxy[1831]: I0818 22:06:00.878182 1831 proxy.go:81] [client #2] hyper(cmd=destroypod, data={})
The content of /run/cc-oci-runtime/cc-oci-runtime.log
is the following.
2017-08-18T16:34:00.968559Z:2041:cc-oci-runtime:debug:cc-oci-runtime 2.2.4 called as: /usr/bin/cc-oci-runtime create --bundle /var/run/docker/libcontainerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b --console /dev/pts/2 --pid-file /run/docker/libcontainerd/containerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/init/pid d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:34:00.968842Z:2041:cc-oci-runtime:debug:path '/var/run/docker/libcontainerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b' resolved to '/run/docker/libcontainerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b'
2017-08-18T16:34:00.968908Z:2041:cc-oci-runtime:debug:using config_file /run/docker/libcontainerd/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/config.json
2017-08-18T16:34:00.973991Z:2041:cc-oci-runtime:debug:path '/var/lib/docker/vfs/dir/e8becfdc6df83ab9f5cb63ff1282b24c6481e56a17f4953390c9f9bfe9bbe685' resolved to '/var/lib/docker/vfs/dir/e8becfdc6df83ab9f5cb63ff1282b24c6481e56a17f4953390c9f9bfe9bbe685'
2017-08-18T16:34:00.974170Z:2041:cc-oci-runtime:debug:Reading VM configuration from /usr/share/defaults/cc-oci-runtime/vm.json
2017-08-18T16:34:00.982747Z:2041:cc-oci-runtime:debug:path '/usr/bin/qemu-lite-system-x86_64' resolved to '/usr/bin/qemu-lite-system-x86_64'
2017-08-18T16:34:00.985866Z:2041:cc-oci-runtime:debug:path '/usr/share/clear-containers/clear-containers.img' resolved to '/usr/share/clear-containers/clear-16670-containers.img'
2017-08-18T16:34:00.985972Z:2041:cc-oci-runtime:debug:path '/usr/share/clear-containers/vmlinux.container' resolved to '/usr/share/clear-containers/vmlinux-4.9.35-62.container'
2017-08-18T16:34:00.986180Z:2041:cc-oci-runtime:debug:OCI spec versions: config=1.0.0-rc2-dev, runtime=1.0.0-rc1
2017-08-18T16:34:00.986285Z:2041:cc-oci-runtime:debug:creating directory /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:34:00.986397Z:2041:cc-oci-runtime:debug:Created rootfs bind mount object for container d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:34:00.986453Z:2041:cc-oci-runtime:debug:Added rootfs bind mount for container d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:34:00.986552Z:2041:cc-oci-runtime:debug:mounting /var/lib/docker/vfs/dir/e8becfdc6df83ab9f5cb63ff1282b24c6481e56a17f4953390c9f9bfe9bbe685 of type bind onto /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload//d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/rootfs with options '' and flags 0x1000
2017-08-18T16:34:00.986649Z:2041:cc-oci-runtime:debug:setting up namespaces
2017-08-18T16:34:00.986787Z:2041:cc-oci-runtime:debug:created mount namespace
2017-08-18T16:34:00.987627Z:2041:cc-oci-runtime:debug:created network namespace
2017-08-18T16:34:00.987690Z:2041:cc-oci-runtime:debug:ignoring uts namespace request
2017-08-18T16:34:00.987738Z:2041:cc-oci-runtime:debug:ignoring pid namespace request
2017-08-18T16:34:00.987785Z:2041:cc-oci-runtime:debug:ignoring ipc namespace request
2017-08-18T16:34:00.987832Z:2041:cc-oci-runtime:debug:finished namespace setup
2017-08-18T16:34:00.987890Z:2041:cc-oci-runtime:debug:Mounting mount /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/31356b6379ce2018-resolv.conf for mnt_dir /etc/resolv.conf
2017-08-18T16:34:00.987963Z:2041:cc-oci-runtime:debug:mounting /var/lib/docker/containers/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/resolv.conf of type bind onto /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/31356b6379ce2018-resolv.conf with options '' and flags 0x45000
2017-08-18T16:34:00.988053Z:2041:cc-oci-runtime:debug:Mounting mount /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/8c6bf0ced6e6fcb7-hostname for mnt_dir /etc/hostname
2017-08-18T16:34:00.988126Z:2041:cc-oci-runtime:debug:mounting /var/lib/docker/containers/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/hostname of type bind onto /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/8c6bf0ced6e6fcb7-hostname with options '' and flags 0x45000
2017-08-18T16:34:00.988223Z:2041:cc-oci-runtime:debug:Mounting mount /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/b27dca258d03970f-hosts for mnt_dir /etc/hosts
2017-08-18T16:34:00.988297Z:2041:cc-oci-runtime:debug:mounting /var/lib/docker/containers/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/hosts of type bind onto /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload/b27dca258d03970f-hosts with options '' and flags 0x45000
2017-08-18T16:34:00.988393Z:2041:cc-oci-runtime:debug:connecting to proxy cc-proxy
2017-08-18T16:34:00.988728Z:2041:cc-oci-runtime:debug:connected to proxy socket /var/run/cc-oci-runtime/proxy.sock
2017-08-18T16:34:00.989047Z:2041:cc-oci-runtime:debug:hypervisor child pid is 2042
2017-08-18T16:34:00.989186Z:2042:cc-oci-runtime:debug:disconnecting from proxy
2017-08-18T16:34:00.989373Z:2042:cc-oci-runtime:debug:reading hypervisor command-line length from pipe
2017-08-18T16:34:00.989357Z:2041:cc-oci-runtime:debug:shim process running with pid 2043
2017-08-18T16:34:00.989500Z:2043:cc-oci-runtime:debug:shim child waiting for proxy socket fd on fd 6
2017-08-18T16:34:00.989735Z:2041:cc-oci-runtime:debug:created state file /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/state.json
2017-08-18T16:34:01.071447Z:2041:cc-oci-runtime:debug:building hypervisor command-line
2017-08-18T16:34:01.071708Z:2041:cc-oci-runtime:debug:Discovering container interfaces
2017-08-18T16:34:01.071794Z:2041:cc-oci-runtime:debug:Interface := [lo]
2017-08-18T16:34:01.071863Z:2041:cc-oci-runtime:debug:Interface := [eth0]
2017-08-18T16:34:01.071929Z:2041:cc-oci-runtime:debug:Interface := [lo]
2017-08-18T16:34:01.072001Z:2041:cc-oci-runtime:debug:Interface := [eth0]
2017-08-18T16:34:01.072095Z:2041:cc-oci-runtime:debug:IP := [172.17.0.2]
2017-08-18T16:34:01.072171Z:2041:cc-oci-runtime:debug:IP := [255.255.0.0]
2017-08-18T16:34:01.072245Z:2041:cc-oci-runtime:debug:MTU for interface eth0: 1500
2017-08-18T16:34:01.072317Z:2041:cc-oci-runtime:debug:Interface := [lo]
2017-08-18T16:34:01.072384Z:2041:cc-oci-runtime:debug:Interface := [eth0]
2017-08-18T16:34:01.072453Z:2041:cc-oci-runtime:debug:IP := [fe80::42:acff:fe11:2]
2017-08-18T16:34:01.072520Z:2041:cc-oci-runtime:debug:netlink_get_default_gw
2017-08-18T16:34:01.072614Z:2041:cc-oci-runtime:debug:table=254
2017-08-18T16:34:01.072687Z:2041:cc-oci-runtime:debug:IP := [172.17.0.1]
2017-08-18T16:34:01.072754Z:2041:cc-oci-runtime:debug:gateway : 172.17.0.1
2017-08-18T16:34:01.072870Z:2041:cc-oci-runtime:debug:ifname=eth0
2017-08-18T16:34:01.072945Z:2041:cc-oci-runtime:debug:table=254
2017-08-18T16:34:01.073015Z:2041:cc-oci-runtime:debug:IP := [172.17.0.0]
2017-08-18T16:34:01.073082Z:2041:cc-oci-runtime:debug:destination : 172.17.0.0
2017-08-18T16:34:01.073160Z:2041:cc-oci-runtime:debug:ifname=eth0
2017-08-18T16:34:01.073231Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073297Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073363Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073428Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073494Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073559Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073624Z:2041:cc-oci-runtime:debug:table=255
2017-08-18T16:34:01.073693Z:2041:cc-oci-runtime:debug:[1] networks discovered
2017-08-18T16:34:01.073760Z:2041:cc-oci-runtime:debug:Running command : iptables-save
2017-08-18T16:34:01.075544Z:2041:cc-oci-runtime:debug:Running commands for dropping iptables
2017-08-18T16:34:01.075645Z:2041:cc-oci-runtime:debug:arg: '/bin/sh'
2017-08-18T16:34:01.075720Z:2041:cc-oci-runtime:debug:arg: '-c'
2017-08-18T16:34:01.075790Z:2041:cc-oci-runtime:debug:arg: 'iptables -P INPUT ACCEPT && iptables -P FORWARD ACCEPT && iptables -P OUTPUT ACCEPT && iptables -t nat -F && iptables -t mangle -F && iptables -F && iptables -X'
2017-08-18T16:34:01.085339Z:2041:cc-oci-runtime:debug:MTU set for interface ceth0 as 1500
2017-08-18T16:34:01.085441Z:2041:cc-oci-runtime:debug:netlink_link_add_bridge beth0
2017-08-18T16:34:01.085510Z:2041:cc-oci-runtime:debug:Turning off multicast snooping for bridge beth0
2017-08-18T16:34:01.085823Z:2041:cc-oci-runtime:debug:netlink_link_set_addr eth0
2017-08-18T16:34:01.085894Z:2041:cc-oci-runtime:debug:macaddr 02:00:ca:fe:00:00
2017-08-18T16:34:01.086026Z:2041:cc-oci-runtime:debug:netlink_link_set_master 2 3
2017-08-18T16:34:01.086212Z:2041:cc-oci-runtime:debug:netlink_link_set_master 8 3
2017-08-18T16:34:01.086402Z:2041:cc-oci-runtime:debug:netlink_link_enable[1] ceth0
2017-08-18T16:34:01.086509Z:2041:cc-oci-runtime:debug:netlink_link_enable[1] eth0
2017-08-18T16:34:01.086584Z:2041:cc-oci-runtime:debug:netlink_link_enable[1] beth0
2017-08-18T16:34:01.086746Z:2041:cc-oci-runtime:debug:network configuration complete
2017-08-18T16:34:01.086860Z:2041:cc-oci-runtime:debug:using /usr/share/defaults/cc-oci-runtime/hypervisor.args
2017-08-18T16:34:01.096717Z:2041:cc-oci-runtime:debug:guest agent ctl socket: /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-ctl.sock
2017-08-18T16:34:01.096784Z:2041:cc-oci-runtime:debug:guest agent tty socket: /var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-tty.sock
2017-08-18T16:34:01.096923Z:2041:cc-oci-runtime:debug:checking child setup (blocking)
2017-08-18T16:34:01.097041Z:2042:cc-oci-runtime:debug:reading hypervisor command-line from pipe
2017-08-18T16:34:01.097234Z:2042:cc-oci-runtime:debug:running command:
2017-08-18T16:34:01.097310Z:2042:cc-oci-runtime:debug:arg: '/usr/bin/qemu-lite-system-x86_64'
2017-08-18T16:34:01.097378Z:2042:cc-oci-runtime:debug:arg: '-name'
2017-08-18T16:34:01.097443Z:2042:cc-oci-runtime:debug:arg: 'e65a82562cfc'
2017-08-18T16:34:01.097507Z:2042:cc-oci-runtime:debug:arg: '-machine'
2017-08-18T16:34:01.097572Z:2042:cc-oci-runtime:debug:arg: 'pc-lite,accel=kvm,kernel_irqchip,nvdimm'
2017-08-18T16:34:01.097638Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.097709Z:2042:cc-oci-runtime:debug:arg: 'nvdimm,memdev=mem0,id=nv0'
2017-08-18T16:34:01.097777Z:2042:cc-oci-runtime:debug:arg: '-object'
2017-08-18T16:34:01.097848Z:2042:cc-oci-runtime:debug:arg: 'memory-backend-file,id=mem0,mem-path=/usr/share/clear-containers/clear-16670-containers.img,size=235929600'
2017-08-18T16:34:01.097917Z:2042:cc-oci-runtime:debug:arg: '-m'
2017-08-18T16:34:01.097981Z:2042:cc-oci-runtime:debug:arg: '2G,slots=2,maxmem=3G'
2017-08-18T16:34:01.098045Z:2042:cc-oci-runtime:debug:arg: '-kernel'
2017-08-18T16:34:01.098110Z:2042:cc-oci-runtime:debug:arg: '/usr/share/clear-containers/vmlinux-4.9.35-62.container'
2017-08-18T16:34:01.098176Z:2042:cc-oci-runtime:debug:arg: '-append'
2017-08-18T16:34:01.098241Z:2042:cc-oci-runtime:debug:arg: 'root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug init=/usr/lib/systemd/systemd systemd.unit=cc-agent.target iommu=off quiet systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket systemd.show_status=false cryptomgr.notests net.ifnames=0 ip=::::::d3b785dd3ec6::off::'
2017-08-18T16:34:01.098339Z:2042:cc-oci-runtime:debug:arg: '-smp'
2017-08-18T16:34:01.098407Z:2042:cc-oci-runtime:debug:arg: '2,sockets=1,cores=2,threads=1'
2017-08-18T16:34:01.098474Z:2042:cc-oci-runtime:debug:arg: '-cpu'
2017-08-18T16:34:01.098538Z:2042:cc-oci-runtime:debug:arg: 'host'
2017-08-18T16:34:01.098603Z:2042:cc-oci-runtime:debug:arg: '-rtc'
2017-08-18T16:34:01.098666Z:2042:cc-oci-runtime:debug:arg: 'base=utc,driftfix=slew'
2017-08-18T16:34:01.098730Z:2042:cc-oci-runtime:debug:arg: '-no-user-config'
2017-08-18T16:34:01.098803Z:2042:cc-oci-runtime:debug:arg: '-nodefaults'
2017-08-18T16:34:01.098868Z:2042:cc-oci-runtime:debug:arg: '-global'
2017-08-18T16:34:01.098933Z:2042:cc-oci-runtime:debug:arg: 'kvm-pit.lost_tick_policy=discard'
2017-08-18T16:34:01.098997Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.099085Z:2042:cc-oci-runtime:debug:arg: 'virtio-serial-pci,id=virtio-serial0'
2017-08-18T16:34:01.099153Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.099217Z:2042:cc-oci-runtime:debug:arg: 'virtconsole,chardev=charconsole0,id=console0'
2017-08-18T16:34:01.099283Z:2042:cc-oci-runtime:debug:arg: '-chardev'
2017-08-18T16:34:01.099338Z:2042:cc-oci-runtime:debug:arg: 'socket,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/console.sock,server,nowait,id=charconsole0,signal=off'
2017-08-18T16:34:01.099391Z:2042:cc-oci-runtime:debug:arg: '-chardev'
2017-08-18T16:34:01.099448Z:2042:cc-oci-runtime:debug:arg: 'socket,id=procsock,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/process.sock,server,nowait'
2017-08-18T16:34:01.099517Z:2042:cc-oci-runtime:debug:arg: '-chardev'
2017-08-18T16:34:01.099582Z:2042:cc-oci-runtime:debug:arg: 'socket,id=charch0,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-ctl.sock,server,nowait'
2017-08-18T16:34:01.099650Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.099714Z:2042:cc-oci-runtime:debug:arg: 'virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0'
2017-08-18T16:34:01.099781Z:2042:cc-oci-runtime:debug:arg: '-chardev'
2017-08-18T16:34:01.099846Z:2042:cc-oci-runtime:debug:arg: 'socket,id=charch1,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-tty.sock,server,nowait'
2017-08-18T16:34:01.099914Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.099977Z:2042:cc-oci-runtime:debug:arg: 'virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1'
2017-08-18T16:34:01.100044Z:2042:cc-oci-runtime:debug:arg: '-uuid'
2017-08-18T16:34:01.100108Z:2042:cc-oci-runtime:debug:arg: '0c55e194-8518-473c-a34a-e65a82562cfc'
2017-08-18T16:34:01.100172Z:2042:cc-oci-runtime:debug:arg: '-qmp'
2017-08-18T16:34:01.100237Z:2042:cc-oci-runtime:debug:arg: 'unix:/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/hypervisor.sock,server,nowait'
2017-08-18T16:34:01.100305Z:2042:cc-oci-runtime:debug:arg: '-nographic'
2017-08-18T16:34:01.100368Z:2042:cc-oci-runtime:debug:arg: '-vga'
2017-08-18T16:34:01.100432Z:2042:cc-oci-runtime:debug:arg: 'none'
2017-08-18T16:34:01.100495Z:2042:cc-oci-runtime:debug:arg: '-netdev'
2017-08-18T16:34:01.100558Z:2042:cc-oci-runtime:debug:arg: 'tap,ifname=ceth0,script=no,downscript=no,id=ceth0,vhost=on'
2017-08-18T16:34:01.100624Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.100688Z:2042:cc-oci-runtime:debug:arg: 'driver=virtio-net-pci,netdev=ceth0,mac=02:42:ac:11:00:02'
2017-08-18T16:34:01.100753Z:2042:cc-oci-runtime:debug:arg: '-device'
2017-08-18T16:34:01.100818Z:2042:cc-oci-runtime:debug:arg: 'virtio-9p-pci,fsdev=workload9p,mount_tag=rootfs'
2017-08-18T16:34:01.100885Z:2042:cc-oci-runtime:debug:arg: '-fsdev'
2017-08-18T16:34:01.100950Z:2042:cc-oci-runtime:debug:arg: 'local,id=workload9p,path=/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/workload,security_model=none'
2017-08-18T16:34:01.113438Z:2041:cc-oci-runtime:debug:child setup successful
2017-08-18T16:34:01.286281Z:2041:cc-oci-runtime:debug:CTL created event: 3
2017-08-18T16:34:01.286408Z:2041:cc-oci-runtime:debug:CTL created event: 1
2017-08-18T16:34:01.286509Z:2041:cc-oci-runtime:debug:communicating with proxy
2017-08-18T16:34:01.286580Z:2041:cc-oci-runtime:debug:sending message (length 447) to proxy socket
2017-08-18T16:34:01.286634Z:2041:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hello","data":{"containerId":"d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b","ctlSerial":"/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-ctl.sock","ioSerial":"/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/ga-tty.sock","console":"/var/run/cc-oci-runtime/d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b/console.sock"}}
2017-08-18T16:36:00.869590Z:2084:cc-oci-runtime:debug:cc-oci-runtime 2.2.4 called as: /usr/bin/cc-oci-runtime delete d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:36:00.871458Z:2043:cc-oci-runtime:critical:failed to read proxy socket fd
2017-08-18T16:36:00.876648Z:2084:cc-oci-runtime:debug:connecting to proxy cc-proxy
2017-08-18T16:36:00.877115Z:2084:cc-oci-runtime:debug:connected to proxy socket /var/run/cc-oci-runtime/proxy.sock
2017-08-18T16:36:00.877245Z:2084:cc-oci-runtime:debug:communicating with proxy
2017-08-18T16:36:00.877297Z:2084:cc-oci-runtime:debug:sending message (length 105) to proxy socket
2017-08-18T16:36:00.877331Z:2084:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"attach","data":{"containerId":"d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b"}}
2017-08-18T16:36:00.877841Z:2084:cc-oci-runtime:debug:proxy msg length: 37
2017-08-18T16:36:00.877893Z:2084:cc-oci-runtime:debug:message read from proxy socket: {"success":true,"data":{"version":1}}
2017-08-18T16:36:00.877993Z:2084:cc-oci-runtime:debug:msg received: {"success":true,"data":{"version":1}}
2017-08-18T16:36:00.878044Z:2084:cc-oci-runtime:debug:communicating with proxy
2017-08-18T16:36:00.878074Z:2084:cc-oci-runtime:debug:sending message (length 58) to proxy socket
2017-08-18T16:36:00.878099Z:2084:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hyper","data":{"hyperName":"destroypod","data":{}}}
I think the issue is with critical:failed to read proxy socket fd
.
Tried running on Ubuntu 16.04 Server and Desktop VMs on same machine. Same error in both.
Thanks for the help.
@chavafg @jcvenegas Hi - any thoughts here? I suspect we have seen the non-existent proxy socket before maybe. @vinay-y for our info, how did you install CC. Presumably following: https://github.com/01org/cc-oci-runtime/blob/master/documentation/Installing-Clear-Containers-on-Ubuntu.md or https://github.com/01org/cc-oci-runtime/blob/master/documentation/Quickstart-Clear-Containers-in-a-Ubuntu-VM.md
oh, and thanks for the pretty comprehensive report @vinay-y :-)
Hi @vinay-y , agree pretty comprehensive report, for what I can see the timeout was caused because the runtime was waiting for the VM started correctly, but seems that never happend and docker called to delete
commmand.
017-08-18T16:34:01.286634Z:2041:cc-oci-runtime:debug:writing message data to proxy socket: {"id":"hello","data":2017-08-18T16:36:00.869590Z:2084:cc-oci-runtime:debug:cc-oci-runtime 2.2.4 called as: /usr/bin/cc-oci-runtime delete d3b785dd3ec686786ab9113f96f5294d447968c79c63fb75d386da96ce180a3b
2017-08-18T16:36:00.871458Z:2043:cc-oci-runtime:critical:failed to read proxy socket fd
- Could you enable cc-proxy debug logs to get more information about it:
sudo sed -i 's,ExecStart=/usr/libexec/cc-proxy,ExecStart=/usr/libexec/cc-proxy -v 3 ,g' /usr/lib/systemd/system/cc-proxy.service
sudo systemctl daemon-reload
sudo systemctl restart cc-proxy
Then run a container again and collect again the cc-proxy
logs.
2. You mentinued you are using VMs, what is the hypervisor you are using?
Hi. @grahamwhaley I used the first one. https://github.com/01org/cc-oci-runtime/blob/master/documentation/Installing-Clear-Containers-on-Ubuntu.md
@jcvenegas A clarification: All the logs I have uploaded are for a physical machine. In addition to testing on this machine I tried on qemu-kvm VMs on the same machine. My goal is to run containers on VMs. But current priority is to get it running on physical machine.
After enabling cc-proxy debug logs, output of journalctl -u cc-proxy.service
is
-- Logs begin at Sat 2017-08-19 00:59:48 IST, end at Mon 2017-08-21 21:47:31 IST. --
Aug 21 21:42:29 SDN-NFV-Server systemd[1]: Stopped Clear Containers Proxy.
Aug 21 21:42:29 SDN-NFV-Server systemd[1]: Started Clear Containers Proxy.
Aug 21 21:42:29 SDN-NFV-Server cc-proxy[5000]: I0821 21:42:29.392438 5000 proxy.go:381] proxy started
Aug 21 21:44:08 SDN-NFV-Server cc-proxy[5000]: I0821 21:44:08.358831 5000 proxy.go:71] [client #1] client connected
Aug 21 21:44:08 SDN-NFV-Server cc-proxy[5000]: I0821 21:44:08.686025 5000 proxy.go:81] [client #1] hello(containerId=d5d517005372ce688e3dc82fd1bb2b0a741abe6c91cb8f5d6032fbb6193bc63a,ctlSerial=/var/run/cc-oci-runtime/d5d517005372ce688e3dc82fd1bb2b0a741abe6c91cb8f5d6032fbb6193bc63a/ga-ctl.sock,ioSerial=/var/run/cc-oci-runtime/d5d517005372ce688e3dc82fd1bb2b0a741abe6c91cb8f5d6032fbb6193bc63a/ga-tty.sock,console=/var/run/cc-oci-runtime/d5d517005372ce688e3dc82fd1bb2b0a741abe6c91cb8f5d6032fbb6193bc63a/console.sock)
The logs for same command I provided in previous comment are probably wrong as cc-proxy.service was not active then. Only cc-proxy.socket was active.
But, this time because of restart cc-proxy
command, cc-proxy.service started and these are correct logs.
Thanks again.
@vinay-y thank you, seems that we did not get enough information from proxy logs :(.
You could you try running the following script in order to detect if your system could a Clear Container:
curl -O https://download.clearlinux.org/current/clear-linux-check-config.sh
bash ./clear-linux-check-config.sh container
Could you verify if is there any qemu instance running ?
pgrep qemu-lite
Also could you provide cpu and memory information from your host ?
@jcvenegas Thanks for this. Maybe this was the problem. This was the output for the check script.
Checking if host is capable of running Clear Linux* OS for Intel® Architecture in a container
SUCCESS: Intel CPU
SUCCESS: 64-bit CPU (lm)
SUCCESS: Streaming SIMD Extensions v4.1 (sse4_1)
SUCCESS: Virtualisation support (vmx)
SUCCESS: Kernel module kvm
SUCCESS: Kernel module kvm_intel
SUCCESS: Nested KVM support
FAIL: Unrestricted guest KVM support
Can you please tell me how to enable unrestricted guest kvm support? My CPU model name : Intel(R) Xeon(R) CPU E5507 @ 2.27GHz What more information do you need? I am pretty sure qemu-lite was running.
@vinay-y Can you try a quick test on that system to confirm that it may not be able to support Clear Containers. Can you try running the following command as root or under sudo
qemu-lite-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
-cpu host -m 256,maxmem=1G,slots=2 -smp 2 \
-no-user-config -nodefaults -rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-kernel /usr/share/clear-containers/vmlinux.container \
-append "reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1 init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false debug" \
-device virtio-serial-pci,id=virtio-serial0 \
-chardev stdio,id=charconsole0 \
-device virtconsole,chardev=charconsole0,id=console0 \
-nographic \
-object memory-backend-file,id=mem0,share,mem-path=/usr/share/clear-containers/clear-containers.img,size=235929600 \
-device nvdimm,memdev=mem0,id=nv0 -no-reboot
If your platform can actually run the VM, you will end up on the command prompt.
You are in rescue mode. After logging in, type "journalctl -xb" to view
system logs, "systemctl reboot" to reboot, "systemctl default" or ^D to boot
into default mode.
Press Enter for maintenance
(or press Control-D to continue):
If this hangs, then please send us the kernel boot log.
Hi guys, I encountered the same issue in my local ubuntu machine.
The error is same: "cc-oci-runtime:critical:failed to read proxy socket fd", no other error reported/
And I tested my machine by above command "bash ./clear-linux-check-config.sh container" as @jcvenegas mentioned, the result is SUCCESS for all items.
I also checked the qemu command as @mcastelino posted, it shows nothing, is there have some issues about my machine?
Please give us some tips, thanks.
Hi @mcastelino, I ran the command you gave and ended on the command prompt without any output. I don't understand the second block is - "rescue mode" block. What is it for?
synerg@SDN-NFV-Server:~$ sudo qemu-lite-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
> -cpu host -m 256,maxmem=1G,slots=2 -smp 2 \
> -no-user-config -nodefaults -rtc base=utc,driftfix=slew \
> -global kvm-pit.lost_tick_policy=discard \
> -kernel /usr/share/clear-containers/vmlinux.container \
> -append "reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1 init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false debug" \
> -device virtio-serial-pci,id=virtio-serial0 \
> -chardev stdio,id=charconsole0 \
> -device virtconsole,chardev=charconsole0,id=console0 \
> -nographic \
> -object memory-backend-file,id=mem0,share,mem-path=/usr/share/clear-containers/clear-containers.img,size=235929600 \
> -device nvdimm,memdev=mem0,id=nv0 -no-reboot
[sudo] password for synerg:
synerg@SDN-NFV-Server:~$
Thanks.
Hi @jovizhangwei , You recently closed an issue here about this. There was a comment that your friend resolved your problem. What did you do? Were you able to run it finally? Thanks.
Hi @vinay-y , I guess that issue is not same as this one, the error is also not same, we didn't found clear root cause of that issue, but probably caused by multi-version installed and misused.
@vinay-y you should not end up on your system's command prompt. Rather you should have ended up inside the container environment. If you ended up on the system command prompt, it means that the VM did not launch. However I am surprised that you did not see any other output.
I think your system (Nehalem) cannot support clear containers as it lacks Unrestricted Guest mode support.
@mcastelino , is there have any simpler way to check the machine lacks Unrestricted Guest mode? I have some issue with @vinay-y , the qemu command didn't print anything.
I ran into the same error on centos7.3, I follow the https://github.com/01org/cc-oci-runtime/blob/master/documentation/Installing-Clear-Containers-on-Centos-7.md to install clear container
the error logs are almost the same. and the result of sh clear-linux-check-config.sh container
is
Checking if host is capable of running Clear Linux* OS for Intel® Architecture in a container
SUCCESS: Intel CPU
SUCCESS: 64-bit CPU (lm)
SUCCESS: Streaming SIMD Extensions v4.1 (sse4_1)
SUCCESS: Virtualisation support (vmx)
SUCCESS: Kernel module kvm
SUCCESS: Kernel module kvm_intel
SUCCESS: Nested KVM support
SUCCESS: Unrestricted guest KVM support
SUCCESS: Kernel module vhost
SUCCESS: Kernel module vhost_net
but the result of sh clear-linux-check-config.sh host
is
Checking if host is capable of running Clear Linux* OS for Intel® Architecture
SUCCESS: Intel CPU
SUCCESS: 64-bit CPU (lm)
SUCCESS: Streaming SIMD Extensions v4.1 (sse4_1)
FAIL: EFI firmware
when I ran command:
qemu-system-x86_64 -machine pc-lite,accel=kvm,kernel_irqchip,nvdimm \
-cpu host -m 256,maxmem=1G,slots=2 -smp 2 \
-no-user-config -nodefaults -rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-kernel /usr/share/clear-containers/vmlinux.container \
-append "reboot=k panic=1 rw tsc=reliable no_timer_check noreplace-smp root=/dev/pmem0p1 init=/usr/lib/systemd/systemd initcall_debug rootfstype=ext4 rootflags=dax,data=ordered dhcp rcupdate.rcu_expedited=1 clocksource=kvm-clock console=hvc0 single iommu=false debug" \
-device virtio-serial-pci,id=virtio-serial0 \
-chardev stdio,id=charconsole0 \
-device virtconsole,chardev=charconsole0,id=console0 \
-nographic \
-object memory-backend-file,id=mem0,share,mem-path=/usr/share/clear-containers/clear-containers.img,size=235929600 \
-device nvdimm,memdev=mem0,id=nv0 -no-reboot
I got no results just like vinay-y.
looking forward to your help
@vinay-y @jiangpengcheng have you ever tried with CC 3.0 [1][2][3]?
[1] https://github.com/clearcontainers/runtime/blob/master/docs/centos-installation-guide.md [2] https://github.com/clearcontainers/runtime/blob/master/docs/ubuntu-installation-guide.md [3] http://download.opensuse.org/repositories/home:/clearcontainers:/clear-containers-3/
@devimc not yet, I will try it now and give feedback
I tried the CC 3.0, and it works as expected.