hyperd
hyperd copied to clipboard
hyperhq
dockerizing hyperd
I'd like to run hyperd inside a docker image for deployment reasons.
I've managed to get it working by adding lots of privileges (that is okay for my scenario):
$ docker run --privileged --net=host -v /dev:/dev --cap-add=ALL -it myimage bash
$ hyperd >/dev/null &
$ hyperctl run -t alpine:3.2 uptime
09:42:41 up ....
The hyperd prints out following errors:
WARN[0000] devmapper: Udev sync is not supported. This will lead to unexpected behavior, data loss and errors. For more information, see https://docs.docker.com/reference/commandline/daemon/#daemon-storage-driver-option
WARN[0000] devmapper: Usage of loopback devices is strongly discouraged for production use. Please use `--storage-opt dm.thinpooldev` or use `man docker` to refer to dm.thinpooldev section.
WARN[0000] devmapper: XFS is not supported in your system. Either the kernel doesnt support it or mkfs.xfs is not in your PATH. Defaulting to ext4 filesystem
INFO[0000] devmapper: Creating filesystem ext4 on device docker-0:69-235-base
INFO[0001] devmapper: Successfully created filesystem ext4 on device docker-0:69-235-base
INFO[0001] Graph migration to content-addressability took 0.00 seconds
WARN[0001] Your kernel does not support swap memory limit.
INFO[0001] Loading containers: start.
INFO[0001] Loading containers: done.
E0406 09:43:16.823246 1 dm.go:193] losetup: /var/lib/hyper/lib/data: failed to set up loop device: Device or resource busy
My Dockerfile is as follows:
FROM ubuntu:16.04
ENV DEBIAN_FRONTEND=noninteractive
ENV GOPATH=/root/go
RUN apt-get update && apt-get upgrade -y
RUN apt-get install -y software-properties-common && \
add-apt-repository -y ppa:longsleep/golang-backports && \
apt-get update && \
apt-get install -y golang-1.8 && \
ln -s /usr/lib/go-1.8/bin/go /usr/local/bin/go
WORKDIR /root/go/src/github.com/hyperhq
RUN apt-get install -y git
RUN git clone https://github.com/hyperhq/hyperd.git hyperd && \
git clone https://github.com/hyperhq/hyperstart.git hyperstart
RUN apt-get install -y \
autotools-dev libdevmapper-dev libsqlite3-dev libvirt-dev automake \
cpio
WORKDIR /root/go/src/github.com/hyperhq/hyperd
#TODO: btrfs is yes, but:
#checking btrfs/ioctl.h, usability... no
#checking btrfs/ioctl.h, presence... no
#checking for btrfs/ioctl.h,... no
RUN apt-get install -y gawk btrfs-tools
RUN ./autogen.sh && \
./configure --without-xen && \
make
RUN cp hyperd /usr/local/bin && \
cp hyperctl /usr/local/bin && \
mkdir -p /etc/hyper && \
cp package/dist/etc/hyper/config /etc/hyper
WORKDIR /root/go/src/github.com/hyperhq/hyperstart
RUN ./autogen.sh && \
./configure && \
make
RUN mkdir -p /var/lib/hyper && \
cp build/kernel /var/lib/hyper && \
cp build/hyper-initrd.img /var/lib/hyper
RUN apt-get install -y \
aufs-tools libvirt-bin qemu-kvm
#TODO: move
#E0406 08:48:28.327679 16 dm.go:222] /bin/sh: 1: dmsetup: not found
RUN apt-get install -y dmsetup
# WARN[0001] Running modprobe nf_nat failed with message: `modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.4.0-66-generic/modules.dep.bin'
RUN apt-get install -y kmod
RUN apt-get install -y linux-image-4.4.0-66-generic
How could I get rid of those errors?
Tried but give up -- It is too hard to operate low level storage functionalities inside docker container.
Is /dev/loop6 and /dev/loop7 used in your host? hyperd devicemapper storage driver uses these two devices by default.
https://github.com/hyperhq/hyperd/blob/master/daemon/storage.go#L108 https://github.com/hyperhq/hyperd/blob/master/storage/defaults.go#L6
Okay, so this seems to work:
docker run -it --privileged --net=host -v /var/lib/hyper myimage hyperd
So: -v /dev:/dev is not needed and also --cap-add=ALL is not (because of --privileged)
IMO this looks very promising, it's so convenient to run hyperd as a docker image!
@matti Did you benchmark your setup? I followed this to create a hyper-in-docker (hid) installation of the 0.8.1 version and it worked fine, but when running a simple benchmark I get very low disk-IO performance. I.e., 'tar xf linux-4.12.5.tar.xz' takes around 15 seconds on host, also around 15 seconds in docker but around 200 seconds on hid. All my setup uses btrfs (host, docker and hyperd).
@neerdoc didn’t do any IO benchmarks. Just compared startup times (ran uptime from alpine image) - from around 6s on GCP, virtualbox to 1.7s with baremetal kabylake 3.8ghz..
Where did you do the benchmark? Running the tar within the container at /var/lib/hyper with the passed through volume will perform vastly differently then in the rest of the container as it will be doing cow operations.
