kind icon indicating copy to clipboard operation
kind copied to clipboard

Cannot use HostToContainer on MacOS

Open fardarter opened this issue 3 years ago • 15 comments

https://github.com/kubernetes-sigs/kind/blob/bdfe777928f675ebcecc838511d173c36ba6d789/site/static/examples/config-with-mounts.yaml#L18

Cannot use HostToContainer on MacOS as per: https://github.com/kubernetes/kubernetes/issues/61058

fardarter avatar Dec 30 '21 10:12 fardarter

https://github.com/kubernetes-sigs/kind/blob/bdfe777928f675ebcecc838511d173c36ba6d789/site/static/examples/config-with-mounts.yaml#L18

Cannot use HostToContainer on MacOS as per: kubernetes/kubernetes#61058

that is kind config not kubernetes config, that configuration works in Mac

aojea avatar Dec 30 '21 11:12 aojea

There is a discussion on the linked thread.

The error I get is identical:

ERROR: failed to create cluster: docker run error: command "docker run --hostname kind-control-plane --name kind-control-plane --label io.x-k8s.kind.role=control-plane --privileged --security-opt seccomp=unconfined --security-opt apparmor=unconfined --tmpfs /tmp --tmpfs /run --volume /var --volume /lib/modules:/lib/modules:ro --detach --tty --label io.x-k8s.kind.cluster=kind --net kind --restart=on-failure:1 --init=false --volume=/path/WithTestData:/WithTestData:ro,rslave --publish=0.0.0.0:80:80/TCP --publish=0.0.0.0:443:443/TCP --publish=0.0.0.0:31433:31433/TCP --publish=127.0.0.1:64174:6443/TCP -e KUBECONFIG=/etc/kubernetes/admin.conf kindest/node:v1.21.1@sha256:69860bda5563ac81e3c0057d654b5253219618a22ec3a346306239bba8cfa1a6" failed with error: exit status 125
Command Output: 80f688eb0b45e1d5e8db561d1748a602581be5c6834a3a2df09a13cf52e43953
docker: Error response from daemon: path /host_mnt//path/WithTestData is mounted on /host_mnt but it is not a shared or slave mount.
make: *** [set-up-kind] Error 1

fardarter avatar Dec 30 '21 11:12 fardarter

The error may be identical , but in this case is docker who fails to mount the volume docker run --hostname ... --volume=/path/WithTestData:/WithTestData:ro,rslave

The kubernetes cluster is still not running, try to test your docker installation without kind , there has to be something there ...

aojea avatar Dec 30 '21 11:12 aojea

Using Docker Desktop for Mac. I've checked the file share paths (as per docs) and all is in order (cluster starts fine if I remove the field).

If you have any ideas I'm all ears.

fardarter avatar Dec 30 '21 12:12 fardarter

we have a template to report issues like this, otherwise is hard to understand the problem, ideally, it is easier if you explain how to reproduce it.

At least we need

  • kind version: (use kind version):
  • Docker version: (use docker info):
  • kind configuration file

aojea avatar Dec 30 '21 17:12 aojea

this is a dupe of https://github.com/kubernetes-sigs/kind/issues/2400

this is not possible with host shared directories on the mac. it is a limitation based on how docker desktop works

BenTheElder avatar Dec 30 '21 22:12 BenTheElder

If you're correct, @BenTheElder, then a comment on the docs would be correct, IMO, especially as they're example docs.

fardarter avatar Dec 31 '21 05:12 fardarter

PRs to the docs warning about this would be most welcome, I think.

The configuration guide has a note that it's still a bit lacking at the top, this field is a pretty "power user" feature and there's just so much to do.

BenTheElder avatar Jan 05 '22 17:01 BenTheElder

PRs to the docs warning about this would be most welcome, I think.

The configuration guide has a note that it's still a bit lacking at the top, this field is a pretty "power user" feature and there's just so much to do.

Normally I'm happy to make docs PRs but I think it needs some explanation about why it doesn't work (even one sentence or a link) and I'm not really sure I get why (docker inner workings are not my thing -- I'm just setting up a k8s dev environment for local and am trying to keep things isolated from host.).

fardarter avatar Jan 05 '22 17:01 fardarter

Roughly mentioned in https://github.com/kubernetes-sigs/kind/issues/2400#issuecomment-889997259

Not in a nice docs comment form but a more detailed brain dump on that:

Mount propagation can't happen because one filesystem is running in the macOS kernel on the host and the other is in the docker desktop linux kernel inside a virtual machine.

These operating system kernels do not share mounts, full stop. Docker doesn't actually make any shared mounts between the host and the virtual machine, that functionality doesn't exist.

Instead it emulates individual shared mounts the user requested when running the container by synchronizing filesystem state between the host and the virtual machine. It hooks filesystem calls within these mounts on the virtual machine side so when you list the directory or file contents inside the VM it intercepts those system calls and responds with results from the macOS host, roughly.

Mount propagation would require that new mounts dynamically created at runtime show up, which wouldn't make sense as the original mount never showed up, instead the contents of the directory are actively mapped between the two kernels as closely as possible.


I think it would be fair to just add a warning along the lines of "mount propagation won't work with the docker desktop app, this is a Docker limitation" and perhaps link to this thread.

If the set the same equivalent options are used just docker run foo the same underlying docker desktop limitation will arise, so I don't think it's this project's place to go into great detail on that. It would be nice if docker covered this, perhaps here https://docs.docker.com/storage/bind-mounts/#configure-bind-propagation

BenTheElder avatar Jan 05 '22 18:01 BenTheElder

OK, I'll do a little PR tomorrow.

fardarter avatar Jan 05 '22 18:01 fardarter

/help /good-first-issue For anyone interested: our contributor guide has details on how to develop the documentation site.

BenTheElder avatar Apr 22 '22 16:04 BenTheElder

@BenTheElder: This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

  • Why are we solving this issue?
  • To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
  • Does this issue have zero to low barrier of entry?
  • How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-good-first-issue command.

In response to this:

/help /good-first-issue For anyone interested: our contributor guide has details on how to develop the documentation site.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot avatar Apr 22 '22 16:04 k8s-ci-robot

any suggested solution ? i am also facing same issues on mac .

Error: failed to start container "provisioner": Error response from daemon: path /Users/****/mysql/data is mounted on /host_mnt but it is not a shared or slave mount

amleshk66 avatar Aug 23 '22 07:08 amleshk66

you cannot use this on macOS if you're using a host mount*. you also probably don't need it. https://github.com/kubernetes-sigs/kind/issues/2576#issuecomment-1005958468

* mounts that are from within the VM can do this. mounts that cross the VM boundary back to macOS cannot.

BenTheElder avatar Aug 23 '22 21:08 BenTheElder

I've added a bit more details to the docs around this.

BenTheElder avatar Oct 20 '22 22:10 BenTheElder