mesosphere
Trying to use Weave as an overlay network
I've been attempting to use Weave as an overlay network with Kubernetes-Mesos, but for some reason whenever I launch tasks, the resulting containers are all created ignoring Weave. I've tried setting the DOCKER_HOST variable to the weave socket, as well as setting the PATH in the executor_environment_variables config file, but things just hang after that. Not sure what my next steps should be. Wondering if Weave is even supported.
Kubernetes v1.2.0-alpha.4.603+f1f250c6b4fffa Mesos 0.27.0 Weave 1.3.1
try setting --host_port_endpoints=false when you run the k8sm-controller-manager. the automatic port mapping in k8sm is probably a bit aggressive for people that don't want to use default docker networking.
also see the notes in the known issues doc related to k8s.mesosphere.io/portMapping
xref #527
Thanks for the suggestion. I've tried that before but it hasn't helped. The issue is that the actual containers are being created on the default Docker bridge on each host, not the Weave network. Docker doesn't let me change the default network, so I was wondering if kubernetes-mesos would.
would it help to be able to use some of the newer kubelet network drivers? e.g. https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/network/kubenet/kubenet_linux.go
On Mon, Feb 8, 2016 at 7:46 AM, Brandon [email protected] wrote:
Thanks for the suggestion. I've tried that before but it hasn't helped. The issue is that the actual containers are being created on the default Docker bridge on each host, not the Weave network. Docker doesn't let me change the default network, so I was wondering if kubernetes-mesos would.
— Reply to this email directly or view it on GitHub https://github.com/mesosphere/kubernetes-mesos/issues/765#issuecomment-181353480 .
If what that does is allow me to tell the executor to use a specific Docker network other than the default, then yes definitely, that's exactly the functionality I'm looking for.
It appears to do some bridge reconfiguration and uses a specific subnet that may be passed in as a command line flag. Basically, it configures the node just like the default salt scripts, only its implemented as a kubelet network driver. So this does let you configure pods to use specific a specific CIDR range for their IPs but it also configures the bridge IP as well (and assumes a specific bridge name). Using this driver, I'm pretty sure that the kubelet tells Docker to use --net:none for the pod infra container and then the network driver takes care of wiring up the networking for the pod on its own.
I'm not sure how you're setting DOCKER_HOST or PATH for the executor that gets run, but it seems to me that you'd want to use the --kubelet-docker-endpoint flag of the k8sm-scheduler to point to the location of the weave socket (should be the same on all slaves).
I was trying to go underneath k8s to Mesos and tell it about Weave, using the executor_environment_variables flag for my slaves. I thought that might take care of the executor using the wrong network, basically if it didn't have a choice.
I'm hesitant to upgrade right now but I think I'll try and test it out off-cluster. I've seen that flag before as well, not sure if it changed things but I'll throw it in and see what happens.
OK, after setting that flag, nothing changed. My containers are still being created on the default Docker net. However, in the executor logs, I think I can see Docker connecting to the Weave socket. That makes me think that Weave is ignoring it's own settings, but when I manually start a container with docker run pointing at the Weave socket, everything works. The executor Mesos task also has the docker-endpoint flag set to the Weave socket, so I'm at a loss here. Everything seems like it should be working, but it's not.
Can you post your executor logs somewhere? Perhaps as a gist or pastebin?
On Mon, Feb 8, 2016 at 10:02 AM, Brandon [email protected] wrote:
OK, after setting that flag, nothing changed. My containers are still being created on the default Docker net. However, in the executor logs, I think I can see Docker connecting to the Weave socket. That makes me think that Weave is ignoring it's own settings, but when I manually start a container with docker run pointing at the Weave socket, everything works. The executor Mesos task also has the docker-endpoint flag set to the Weave socket, so I'm at a loss here. Everything seems like it should be working, but it's not.
— Reply to this email directly or view it on GitHub https://github.com/mesosphere/kubernetes-mesos/issues/765#issuecomment-181402053 .
Here's my executor.log: http://pastebin.com/LSpshmxw Here's my proxy.log: http://pastebin.com/4J1CGr8z Here's my stderr: http://pastebin.com/gVBPKLQi
you may be interested in #785 (which should be merged to master shortly); i've also spent some time trying to get this working. relatively successful with the changes from #785 : https://gist.github.com/jdef/55067cd96dcd5302dd11
I managed to get Mesos to use Weave by setting "docker_socket" to Weave's socket instead of Docker's. This only worked on the latest versions of Docker and Weave and Mesos 0.28.0. A slightly older version of Weave also resulted in an overall hang.
