docker-machine-driver-xhyve
docker-machine-driver-xhyve copied to clipboard
machine-drivers
Reporting for launched xhyve
This thread is not issue thread. Post a report only. See also Would you do me a favor?
OK, Running, Good, etc... Anything will do.
Hey, I would like to try the driver, but it seems to have a lot of dependencies https://github.com/zchee/docker-machine-xhyve#libguestfs (OSX Fuse, etc.). Mind uploading a binary?
@johanneswuerbach Thanks issue!
I can upload the binary, but the binary requests a libguestfs.dylib as a dynamic linking.
The reason, OS X does not support static build.
If remove the libguestfs, there are several problems.
- Needs way to copy the
userdata.tarto disk image before launch the xhyve - Give up use
extfsDocker Backing Filesystem (will be usingtmpfs. bad performance)
I know how to solve these without libguestfs, but I felt that pretty way used libguestfs.
If you are satisfied with tmpfs, I will make use tmpfs version binary from now.
BTW, I also have a problem in many of dependency. I'm looking for a better way to make ext.4 disk image without a libguestfs.
Sorry terrible English :(
I definitely see why you used libguestfs, but can't b2d auto-format the disk when it discovers an unformatted one? https://github.com/boot2docker/boot2docker/blob/master/rootfs/rootfs/etc/rc.d/automount#L13-L16
@johanneswuerbach I tried before. I don't remember so much...
But, docker-machine can not first connect the vm use ssh in that case.
docker-machine core implementation is create id_rsa, id_rsa, *.pem based on $HOME/.docker/machine/certs/* before launch any vm.
And, docker also docker-machine is doing a secure remote connection use *.pem.
So, need to copy the locally created userdata.tar in the first.
Looks like b2d is just dd-ing a tar on that specific partition containing the certs: https://github.com/boot2docker/boot2docker/blob/master/rootfs/rootfs/etc/rc.d/automount#L22
Maybe the xhyve driver could create the same using hdiutil and dd.
@johanneswuerbach I remembered.
Yes. The method was previously used.
docker-machine boot2docker provisioner requires ssh login to some of the setting. e.g. https://github.com/docker/machine/blob/master/libmachine/provision/boot2docker.go#L117-L127
If not exist /var/lib/boot2docker/userdata.tar, boot2dokcer is automatically formatted own disk, and create *.pem.
and, did not write hostid_rsa.pub data into authorized_keys.
In that case, It means for ssh login is needs tcuser password.
create blank disk image use dd
|
boot2docker automatically create some files
|
ssh login use tcuser password
|
delete *.pem, and copy the local pem to img. and edit authorized_keys
I knew this way, but I feel it's pushy.
Hmm... Let me think about it.
I've been trying for a while to get this working, but it fails to find the -F flag (introduced in an unmerged PR against xhyve):
$ docker-machine -D create -d xhyve xhyve-dev
...
(xhyve-dev) DBG | operation not supported by device
(xhyve-dev) DBG | xhyve: illegal option -- F
(xhyve-dev) DBG | Usage: xhyve [-behuwxMACHPWY] [-c vcpus] [-g <gdb port>] [-l <lpc>]
(xhyve-dev) DBG | [-m mem] [-p vcpu:hostcpu] [-s <pci>] [-U uuid] -f <fw>
...
The full output: debug.txt
It's working for me! Only thing was for some reason it didn't set up the boot script for the XHYVE_EXPERIMENTAL_NFS_SHARE, so I had to manually ssh in and mount the NFS share. After doing that, it's cranking away on building Swift from source now!
@rothomp3 Thanks a report!!
and, Sure thing for nfs issue. I will create a issue threads, and debugging after fix already exists some of the issue :) Could you provide us with information in order to solve the problem?
(xhyve-dev) DBG | Not there yet 1/60, error: IP not found for MAC 4a:db:63:46:cf:32 in DHCP leases
(xhyve-dev) DBG | operation not supported by device
(xhyve-dev) Waiting on a pseudo-terminal to be ready... virtio_net: Could not create vmnet interface, permission denied or no entitlement?
(xhyve-dev) DBG | exit status 1 &{0xc8200140f0} &{0xc820014120}
...
(xhyve-dev) DBG | Not there yet 60/60, error: IP not found for MAC 4a:db:63:46:cf:32 in DHCP leases
Error creating machine: Error in driver during machine creation: Machine didn't return an IP after 120 seconds, aborting
@Jaesin Hi,
Could you try this command and post result?
ls -la $(which -a docker-machine-driver-xhyve)
I just wiped it out and started again and got it to work. 3rd try is a charm!
Thanks for the effort @zchee
docker-machine ssh xhyve-dev
(xhyve-dev) Getting to VM state...
## .
## ## ## ==
## ## ## ## ## ===
/"""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ / ===- ~~~
\______ o __/
\ \ __/
\____\_______/
_ _ ____ _ _
| |__ ___ ___ | |_|___ \ __| | ___ ___| | _____ _ __
| '_ \ / _ \ / _ \| __| __) / _` |/ _ \ / __| |/ / _ \ '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__| < __/ |
|_.__/ \___/ \___/ \__|_____\__,_|\___/ \___|_|\_\___|_|
Boot2Docker version 1.9.1, build master : cef800b - Fri Nov 20 19:33:59 UTC 2015
Docker version 1.9.1, build a34a1d5
docker@boot2docker:~$
Didn't use the NFS flag as I already have some NFS stuff setup.
Worked brilliantly with Boot2Docker - b2d image 1.9.1
Didn't work with RancherOS because the ROS image has a vmlinuz file, not a vmlinuz64 file - ROS image 0.4.2
When it failed with ROS, it didn't clean up after itself (left the mount point).
Within B2D it was reporting it had 17GiB ram when running top .
@far-blue Thanks, issue :) and reporting. but, I am not good English :( So, I do not yet understand all.
Do you want to run the RancherOS use docker-machine-driver-xhyve?
Within B2D it was reporting it had 17GiB ram when running top.
ram...? In fact, that means available B2D memory size?
I just installed and tested xhyve-docker-machine-driver on OS X El Capitan, docker machine 0.5.6, and it worked brilliantly from the first try with a fairly complex project involving:
- running 3 containers (1 mongodb, 1 api server and 1 web app on nginx)
- using a bridge network
- mounting host files and directories using the experimental nfs share
- mounting the docker socket as a volume in the server container
- building docker images inside the server container
- and running the generated image
So big thanks to @zchee (and the xhyve project too) and keep up the good work !
@zchee:
Do you want to run the RancherOS use docker-machine-driver-xhyve?
Yes, RancherOS (https://github.com/rancher/os) has support for docker-machine as an alternative for Boot2Docker and it provided support for xhyve early on. This is the command I used:
docker-machine create --driver xhyve --xhyve-boot2docker-url https://github.com/rancher/os/releases/download/v0.4.2/rancheros.iso dev
ram...? In fact, that means available B2D memory size?
When I ssh into boot2docker (docker-machine ssh dev) and run top I receive the following:
Tasks: 93 total, 1 running, 92 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.7/0.0 1
GiB Mem : 17.2/0.973
GiB Swap: 0.0/0.207
This might be something best reported to the xhyve project but I just wanted to mention it :)
@far-blue I understand :)
RancherOS provision does not support Now. Currently, boot2docker only :( but, I was planning to support CoreOS and RancherOS provisioning from previous.
However, The first goal is to work perfectly the boot2docker. I think boot2docker is a standard specification of use the docker in OSX. What do you think?
BTW, If the PR does not come, I want to start with its implementation in about next month.
@jawher Thanks for Detailed reporting of operations!! This report is good references to me.
I’m going to give it everything I’ve got.
@zchee:
RancherOS provision does not support Now. Currently, boot2docker only :(
Given RancherOS has worked with xhyve in the past (https://github.com/rancher/os/issues/379) my guess is that you just need to modify the following line to check for either 'vmlinuz64' or 'vmlinuz':
https://github.com/zchee/docker-machine-driver-xhyve/blob/d61209a16e2082074a146da646d3e7269da94ab9/xhyve/xhyve.go#L710
However, you are right, it is best to focus on Boot2Docker first because it is the default and most widely used configuration :)
And thank you for all the work you've done so far - it is looking great!
I followed the guide in https://allysonjulian.com/setting-up-docker-with-xhyve/ and it worked like a charm.
However when I then started setting up a project to use vagrant, with docker as the provider, it wanted to download the virtualbox box as a host rather than my local dev docker machine.
I've limited experience so I'd love a pointer in the right direction.
Thanks for the project.
Great that you created this driver, thanks! 🙌🏼 I wanted to use xhyve in a Docker context for some time but didn't find the time to get familiar with the tooling and automation.
I installed docker, docker-machine, xhyve and docker-machine-driver-xhyve using Homebrew. Running the following doesn't succeed:
$ docker-machine create xhyve-vm -d xhyve
Running pre-create checks...
Creating machine...
(xhyve-vm) Copying /Users/lenn4rd/.docker/machine/cache/boot2docker.iso to /Users/lenn4rd/.docker/machine/machines/xhyve-vm/boot2docker.iso...
(xhyve-vm) Creating VM...
(xhyve-vm) Extracting vmlinuz64 and initrd.img from boot2docker.iso...
(xhyve-vm) /dev/disk5 /Users/lenn4rd/.docker/machine/machines/xhyve-vm/b2d-image
(xhyve-vm) "disk5" unmounted.
(xhyve-vm) "disk5" ejected.
(xhyve-vm) Generating 20000MB disk image...
(xhyve-vm) created: /Users/lenn4rd/.docker/machine/machines/xhyve-vm/root-volume.sparsebundle
(xhyve-vm) Creating SSH key...
(xhyve-vm) Fix file permission...
(xhyve-vm) Generate UUID...
(xhyve-vm) Convert UUID to MAC address...
Error creating machine: Error in driver during machine creation: exit status 1
I remember that xhyve has or had some issues creating network interfaces unless it is executed as root, so I tried sudo:
# sudo docker-machine create xhyve-vm -d xhyve
Password:
Running pre-create checks...
Creating machine...
(xhyve-vm) Copying /Users/lenn4rd/.docker/machine/cache/boot2docker.iso to /Users/lenn4rd/.docker/machine/machines/xhyve-vm/boot2docker.iso...
(xhyve-vm) Creating VM...
(xhyve-vm) Extracting vmlinuz64 and initrd.img from boot2docker.iso...
(xhyve-vm) /dev/disk6 /Users/lenn4rd/.docker/machine/machines/xhyve-vm/b2d-image
(xhyve-vm) "disk6" unmounted.
(xhyve-vm) "disk6" ejected.
(xhyve-vm) Generating 20000MB disk image...
(xhyve-vm) created: /Users/lenn4rd/.docker/machine/machines/xhyve-vm/root-volume.sparsebundle
(xhyve-vm) Creating SSH key...
(xhyve-vm) Fix file permission...
(xhyve-vm) Generate UUID...
(xhyve-vm) Convert UUID to MAC address...
(xhyve-vm) Starting xhyve-vm...
(xhyve-vm) Waiting for VM to come online...
(xhyve-vm) Waiting on a pseudo-terminal to be ready... done
(xhyve-vm) Hook up your terminal emulator to /dev/ttys004 in order to connect to your VM
Waiting for machine to be running, this may take a few minutes...
(xhyve-vm) Getting to VM state...
Machine is running, waiting for SSH to be available...
Detecting operating system of created instance...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect Docker to this machine, run: docker-machine env xhyve-vm
I had to try a couple of times to chown the entire VM, enable it unsuccessfully, remove it and re-create it and on the third try I'm able to use it:
$ eval $(docker-machine env xhyve-vm)
$ docker-machine ls
NAME ACTIVE URL STATE URL SWARM DOCKER ERRORS
xhyve-vm * xhyve Running tcp://192.168.64.4:2376 v1.9.1
I was able to do a quick test as per @dwijnand's blog post.
Is there an easier way not using sudo? Or am I missing a step? I remember reading about code-signing the xhyve binary to be able to use it without sudo but I haven't tried this yet.
@lenn4rd Did you do the
$ sudo chown root:wheel /usr/local/bin/docker-machine-driver-xhyve
$ sudo chmod u+s /usr/local/bin/docker-machine-driver-xhyve
steps from the release notes? https://github.com/zchee/docker-machine-driver-xhyve/releases/tag/v0.2.1
@dwijnand No, I didn't see them because I didn't use one of those releases but the Homebrew formula that @zchee created. Due to security concerns the Homebrew maintainers don't allow neither chmoding installed binaries nor including the steps you mentioned in the caveat section.
I now tried again with chown and chmod as you suggested and the VM is then created as expected. Thanks!