docs: reflect new way of configuring flannel with CNI

[peebs]

Explained in this comment: https://github.com/coreos/coreos-kubernetes/pull/715#issuecomment-252355541

[pop]

@pbx0 I'd be up for resolving this if it's still an issue. Do you think you can elaborate on the problem & point me in the direction of any relevant stuff I should read to resolve this?

[peebs]

@ElijahCaine Sure, that would be great if you'd like to do it. Let me try to provide the context, but, bear with me because explaining the CNI jargon, can get a little confusing.

Mainly I filed the issue because prior to https://github.com/coreos/coreos-kubernetes/commit/7ad134744b75d0a8a75af165afeb107ea245f87a we did not use a kubernetes network plugin at all. We directly configured docker to use the flannel network by default for all pods. The kubernetes CNI plugin was optional and only used to setup Calico, which allows more complex control of the overlay network.

However, since https://github.com/coreos/coreos-kubernetes/commit/7ad134744b75d0a8a75af165afeb107ea245f87a we always use the CNI kubernetes network plugin by passing it a default a configuration that just uses plain flannel. This is nice because it works for both rkt and docker without having to change anything. Calico is still optional and uses a different cni configuration.

Thus, when the docs from #715 say ### Set Up the CNI config (optional) that isn't quite true because CNI is not optional anymore and the docs should reflect the new way in which we setup flannel (via CNI).

I'm not sure how far down this path you want to go as our setup scripts continue to diverge overtime from the manual docs and its difficult to keep them in sync as the setup scripts support more features and configurations by default. When rkt support was added to the setup scripts for example, it included quite a few sections of if RUNTIME=rkt setup this thingy which I think would make the manual docs more confusing and error prone then they already are. The question I have is: Should the manual docs should remain a first class k8s setup method or if it should just be an educational introduction to kubernetes that will likely help with future debugging efforts?

[pop]

Whoa. Well that is a whole bunch of complicated I wasn't expecting.

I'll dig around this issue and see if it's worth it to update the manual doc to more accurately reflect the new world order in the scripts. If not I'll talk with some people about maybe migrating this doc toward an educational focus in favor of being an actual instruction manual, since that seems like it might be more sane as we develop things going forward.

In the meantime I'll definitely figure out a way to reflect the less-than-optional state of CNI.

Thanks!

[robszumski]

Should the manual docs should remain a first class k8s setup method or if it should just be an educational introduction to kubernetes that will likely help with future debugging efforts?

I do think they should remain first class, but I understand it adds burden to keep them up to date. A possible solution to this to reflect the ideal Kubernetes + CoreOS cluster: using CNI, full TLS between components, auto-updates on, etc.

[peebs]

@robszumski Sounds fair to me. I'll try to do a better job of migrating changes to the generic scripts into the docs.

One hurdle that will remain troublesome here is testing the manual docs. By definition, testing is... manual which will never be done on a PR basis. Ideally the generic scripts do exactly what the manual setup would have you do anyway and that is easy to test regularly. We just need a better way to keep things in sync properly. Maybe when we move to self-hosting we can take the time to revamp things a bit. That + tls bootstrapping will simplify many aspects of the setup. Not sure how it would look yet though for the manual guides.

[peebs]

Someone ran into this problem here: https://github.com/coreos/coreos-kubernetes/issues/761.

[renoirb]

I volunteer in helping docs, and work on salt states and scripts. I must say it's confusing to see what works, what's broken.

It's confusing because docs mentions the same need more than once. But with different components.

I understand k8s has to support a lot stuff at every levels: OS (CoreOS, RHEL, Fedora, Debian, Hypriot, Ubuntu), Networking (HostOnly, Flannel, VXLAN, Weave), Runtime (Moby, Docker, Rkt), Config management (Juju, SaltStack, Bash).

Can we get a list of what's still valid (i.e. Not "bitrot") in docs at 1.6 (if it's still what's recommended to install). Then some could volunteer helping with what they have at hand and what they're comfortable with.

Maybe we could have a wiki page to list and organize.

[renoirb]

Oops, I thought this was for k8s at large. I'm focussing on arm64 multi-node network. CoreOS seem to not be ARM 64 compatible. Sorry for the noise.