Use case documentation needed: provisioning VM using libvirt

[Vanuan]

What do I need first? Do I need to write a playbook?

I'm looking to setup something like this:

It looks like currently infrakit supports CLI-based deployment using playbooks. Do I get it right?

So if I somehow create a playbook, I would be able to provision a libvirt-based instance?

[Vanuan]

First of all, I think I need to start libvirt plugin, right? But it looks like it isn't available for some reason:

infrakit plugin start
INFO[10-11|04:02:06] config                                   module=cli/plugin url= fn=github.com/docker/infrakit/cmd/infrakit/plugin.Command.func2
Plugins available:
KIND                	EXEC
aws                 	inproc
combo               	inproc
digitalocean        	inproc
docker              	inproc
enrollment          	inproc
file                	inproc
gc                  	inproc
google              	inproc
group               	inproc
hyperkit            	inproc
ibmcloud            	inproc
image               	inproc
ingress             	inproc
inventory           	inproc
kubernetes          	inproc
maas                	inproc
manager             	inproc
oneview             	inproc
oracle              	inproc
packet              	inproc
pool                	inproc
rackhd              	inproc
resource            	inproc
selector/spread     	inproc
selector/tiered     	inproc
selector/weighted   	inproc
simulator           	inproc
swarm               	inproc
tailer              	inproc
terraform           	inproc
time                	inproc
vagrant             	inproc
vanilla             	inproc
vars                	inproc
vsphere             	inproc

This led me to assuming that this function didn't run for some reason: https://github.com/docker/infrakit/blob/a10d75ee2d0a8992edf415630a47b0633b04b0b0/pkg/run/v0/libvirt/libvirt.go#L33-L35

[Vanuan]

After trying to set a breakpoint it says No source file named pkg/run/v0/libvirt/libvirt.go.

So it means it isn't imported anywhere, right?

So I went to see where all providers are imported and look what I found:

https://github.com/docker/infrakit/blob/a10d75ee2d0a8992edf415630a47b0633b04b0b0/cmd/infrakit/providers.go#L5-L22

Git blame led to this: https://github.com/docker/infrakit/pull/872 So libvirt isn't built by default?

Looking back found this: https://github.com/docker/infrakit/pull/718 Along with instructions on how to build it with libvirt.

Apparently there are some issues with go buildings for libvirt. README of this package says it requires libvirt-dev package installed. And in runtime it requires libvirt.so even if you don't use it. Authors suggest to provide libvirt as a separate binary: https://github.com/libvirt/libvirt-go/issues/8

It probably makes sense to make plugins/providers actually installable as separate binaries. Not sure whether it is designed that way.

[Vanuan]

So, this is how I build infrakit with libvirt provider:

sudo apt-get install libvirt-dev
GO_BUILD_TAGS="libvirt builtin providers" make build/infrakit

And it works:

$ infrakit plugin start libvirt
...
$ infrakit plugin ls
INTERFACE           NAME                          LISTEN
Instance/0.6.1      libvirt/default               /home/john/.infrakit/plugins/libvirt

Now I need to figure out how to provide libvirt URI and then - how to start some instances, and then - how to configure those instances.

[Vanuan]

It was quite hard to find. To provide libvirt URIs we use this:

https://github.com/docker/infrakit/blob/a10d75ee2d0a8992edf415630a47b0633b04b0b0/pkg/run/v0/libvirt/libvirt.go#L26

For example:

$ export INFRAKIT_LIBVIRT_URIS=default=qemu+ssh://server/system
$ infrakit plugin start libvirt

To start instances the command line is this:

$ infrakit local libvirt/default provision -y libvirt.yml

Where libvirt.yml looks like this:

Properties:
  MetadataStoragePool: default
  Domain:
  Domain:
    Type: kvm
    Memory:
      Unit: GiB
      Value: 1
    VCPU:
      Value: 1
    OS:
      Type:
        Arch: x86_64
        Machine: pc-i440fx-2.8
        Type: hvm
      Initrd: ./initrd.img
      Kernel: ./bzImage
      KernelArgs: console=ttyS0
      BIOS:
        UseSerial: 'yes'
        RebootTimeout: '1000'
      Boot:
        Dev: hd
    Devices:
      Interfaces:
      - Type: bridge
        Source:
          Bridge: virbr0
        Model:
          Type: virtio
      Serials:
      - Type: pty
      Consoles:
      - Type: pty
        Target:
          Type: serial
          Name: "0"
      Channels:
      - Type: unix
        Target:
          Type: virtio
          Name: org.qemu.guest_agent.0
    OnCrash: destroy
    OnPoweroff: destroy
    OnReboot: restart

Now I need to figure out how to generate those kernel and initrd files.

[Vanuan]

It looks like linuxkit can generate those files. Installed linuxkit like this:

wget https://github.com/linuxkit/linuxkit/releases/download/v0.6/linuxkit-linux-amd64 -O ~/bin/linuxkit
chmod +x ~/bin/linuxkit

Running it like this

linuxkit build --format kernel+initrd linuxkit.yml

where linuxkit is here:

https://github.com/linuxkit/linuxkit/blob/master/linuxkit.yml

Unfortunately, kernel 4.14.74 causes kernel panic when running in qemu. So I switched to 4.4.159

Now I need to solve the next problem: how to build those images or copy them to the remote libvirtd host (when using INFRAKIT_LIBVIRT_URIS=default=qemu+ssh://server/system). Maybe a simple NFS share for now?

[thebsdbox]

We could perhaps extend the functionality from here https://github.com/docker/infrakit/tree/master/pkg/x/remoteboot ?

[Vanuan]

Not sure I understand. remoteboot looks like a PXE server. How does it help?

Currently I'm struggling to find steps needed to provision kvm instances according to some yaml file. And configure those instances with docker swarm/kubernetes.

Here's a list of problems I found along the way (some of them solved):

1. libvirt is not builtin to infrakit
2. providing libvirt remote URI is not documented (INFRAKIT_LIBVIRT_URIS)
3. example libvirt.yaml doesn't work
4. you have to build kernel and initrd using thirdparty tool (at least linuxkit integration is not documented)
5. you have to copy kernel and initrd on remote host is using remote libvirt URI (at least didn't find a way to copy automatically)
6. video device is not working for some reason
7. can't figure out how to specify random device (https://github.com/libvirt/libvirt-go-xml/issues/25)
8. init section doesn't work (something with permissions for "metadata volume")
9. figuring out networking (should I use portforwarding, ip routing or something else to reach VMs)
10. what's linuxkit configuration to build images with docker installed
11. how to setup container orchestration (say, I want 3 manager and 7 worker VMs), what flavors/groups should I use
12. should I reach container orchestrator directly or through infrakit
13. should I install loadbalancer on metal, VM or container

I'm currently on step 7. It looks like something to do with libvirtxml (https://github.com/libvirt/libvirt-go-xml/issues/25)

[Vanuan]

BTW VNC instead of video device working fine (problem 6):

      Graphics:
      - Type: "vnc"
        AutoPort: "yes"
        Port: -1
        Listen: "127.0.0.1"
        Listeners:
          - Type: "address"
            Address: "127.0.0.1"

[Vanuan]

Ok, the problem with RNG is that vendored copy isn't updated for more than a year: https://github.com/libvirt/libvirt-go-xml/commit/7ea44658ebbc28b44d53372502137ecd60caa975