NetworkPolicy support

[phcollignon]

Hello, Is there any support for network policies in kind ? (calico CNI plugin or any other network plugin with NetworkPolicy support ?) Does kindnet support it ? is it the default network plugin ? Thanks

[aojea]

@phcollignon I think that some users reported in the channel that they had network policies working in kind using calico. kindnet is very simple and doesn't have Network Policy support and is not really something in scope. You can replace kindnet with any of the other CNI plugins, just use DisableDefaultCNI: true in your kind configuration and install your own plugin following their instructions once kind finish to create the cluster.

[phcollignon]

Thanks, I'll give it a try. And if I succeed I'll document it ..

[kfyharukz]

Related to the above discussion...

Calico can deploy only to use network policy (i.e. policy-only mode). However, combining PTP with Calico has a problem. Calico expects veth names of Pods are generated with the following algorithm:

func generateHostVethName(prefix, namespace, podname string) string {
		h := sha1.New()
		h.Write([]byte(fmt.Sprintf("%s.%s", namespace, podname)))
		return fmt.Sprintf("%s%s", prefix, hex.EncodeToString(h.Sum(nil))[:11])
}

We have been successfully using Calico NetworkPolicy with a modified plugin. Our example is here. So, how about applying a modified PTP CNI plugin to kindnet?

[phcollignon]

Thanks .. for advice. I'll try it. My main usage is Kubernetes for DEV on local laptop.
I was used to kubeadm-dind-cluster project (K8s docker in docker) which is retired project. So I came to kind but I am wondering if it is a good option for local dev with all features (storage, networkpolicies, ...)
I want to avoid minikube because it's too heavy

[aojea]

Related to the above discussion...

Calico can deploy only to use network policy (i.e. policy-only mode). However, combining PTP with Calico has a problem. Calico expects veth names of Pods are generated with the following algorithm:

func generateHostVethName(prefix, namespace, podname string) string {
		h := sha1.New()
		h.Write([]byte(fmt.Sprintf("%s.%s", namespace, podname)))
		return fmt.Sprintf("%s%s", prefix, hex.EncodeToString(h.Sum(nil))[:11])
}

We have been successfully using Calico NetworkPolicy with a modified plugin. Our example is here. So, how about applying a modified PTP CNI plugin to kindnet?

Kindnet is using the standard CNI plugins https://github.com/containernetworking/plugins

[BenTheElder]

I was used to kubeadm-dind-cluster project (K8s docker in docker) which is retired project. So I came to kind but I am wondering if it is a good option for local dev with all features (storage, networkpolicies, ...)

Storage has a tracking issue and is a bit thorny, we're working with sig-storage upstream. It's possible to deploy something today, but multi-node local storage options for testing are pretty limited at the moment for the entire ecosystem, we aim to fix that. for single node the CSI local hostpath driver should work, or the rancher local storage system.

currently what we ship is roughly the same as what kubeadm-dind-shipped is the same as what hack/local-up-cluster.sh ships...

Network policies can be used also by simply disabling the built in CNI and deploying your preferred CNI. we don't currently ship them as they don't seem to be widely depended on including for kubernetes's own testing, but we might in the future.

So, how about applying a modified PTP CNI plugin to kindnet?

AFAIK GKE also supports containerd nodes using upstream ptp, and has a calico network policy only option ... perhaps we only need to tweak some config.

I'd prefer to get this working with the upstream ptp plugin than need to ship our own.

[kfyharukz]

Indeed, modifying CNI PTP plugin may not be the best way. Let me describe considered options to use Calico network policy:

1. Send a patch to the upstream CNI PTP
2. Edit veth name in Calico's WorkloadEndpoint resource from outside Calico's control
3. Patch PTP locally for kind (I've proposed above)

I did not think option 1 would be accepted by CNI because CNI does not define how to obtain Kubernetes Pod NamespacedName information. It is merely a k8s convention for CNI (shown in this issue).

I think option 3 is a reasonable choice.

[aojea]

Indeed, modifying CNI PTP plugin may not be the best way. Let me describe considered options to use Calico network policy:

1. Send a patch to the upstream CNI PTP
2. Edit veth name in Calico's WorkloadEndpoint resource from outside Calico's control
3. Patch PTP locally for kind (I've proposed above)

I did not think option 1 would be accepted by CNI because CNI does not define how to obtain Kubernetes Pod NamespacedName information. It is merely a k8s convention for CNI (shown in this issue).

I think option 3 is a reasonable choice.

@kfyharukz I think that the problem that you are describing is a Calico feature, not a Kubernetes feature.

From Calico's documentations

Endpoints Calico network policies apply to endpoints. In Kubernetes, each pod is a Calico endpoint. However, Calico can support other kinds of endpoints. There are two types of Calico endpoints: workload endpoints (such as a Kubernetes pod or OpenStack VM) and host endpoints (an interface or group of interfaces on a host).

Calico's Kubernetes network policies work with kindnet, however, Calico has more features, like WorkloadEndpoint that doesn't :).

In order to use Calico's only network policies you should replace kindnet and use Calico as the CNI., disabling kindnet in the kind config file and installing Calico once kinds create the cluster.

kind: Cluster
apiVersion: kind.sigs.k8s.io/v1alpha3
networking:
  disableDefaultCNI: true

[phcollignon]

Little feedback to confirm that NetworkPolicy is supported with Calico CNI. I disabled kindnet and configured Calico by adding this to the config file :

networking:
  disableDefaultCNI: true 
  podSubnet: 192.168.0.0/16 

Then install Calico and disable rpf check (not sure it is still needed):

kubectl apply -f https://docs.projectcalico.org/v3.8/manifests/calico.yaml
kubectl -n kube-system set env daemonset/calico-node FELIX_IGNORELOOSERPF=true

Then I started the cluster again and checked some NetworkPolicies which seem to work as excpeted.

kind create cluster --config configfile

Thanks to all of you for the hints

[aojea]

/close

seems this issue was already solved https://github.com/kubernetes-sigs/kind/issues/842#issuecomment-554775260

[k8s-ci-robot]

@aojea: Closing this issue.

In response to this:

/close

seems this issue was already solved https://github.com/kubernetes-sigs/kind/issues/842#issuecomment-554775260

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

we may or may not want to target testing the network policy API in kubernetes e2e testing anyhow. this is still something to think about.

[BenTheElder]

I'll be revisiting this this quarter, looking at increasing kubernetes test coverage.

[aojea]

this topic is interesting for sig-network, there is also a group to work in a possible network policy v2. let me ask @jayunit100 if he is using KIND, he performed a nice demo the other day in the sig-networking meeting

[jayunit100]

is this issue a little stale ? not sure ... anyways its easy to run kind w network policy support via just installing CNI manually... you can: git clone https://github.com/jayunit100/k8sprototypes/blob/master/kind/kind-local-up.sh and run the kind-local-up script, it will install calico, antrea, or cillium for you (default no args = calico) and that will support networkpolices

[evankanderson]

Nifty, thanks for the link!

Would it make sense to extend the networking stanza to support a CNI install (possibly by running a Job in the kube-system namespace, if YAML manifests alone aren't sufficient)? It would be nice to be able to have declarative one-command YAML files for testing different CNI configurations.

[aojea]

Nifty, thanks for the link!

Would it make sense to extend the networking stanza to support a CNI install (possibly by running a Job in the kube-system namespace, if YAML manifests alone aren't sufficient)? It would be nice to be able to have declarative one-command YAML files for testing different CNI configurations.

If you mean to add an option within kind to say

networking:
  CNI: Calico|Antrea|Weave|Kindnet|OVN

that is unmaintainable on the long term, each CNI have their own issues and not all of them work in all environments, and we'll end with an incomprehensible matrix of kubernetes/CNI-X versions.

Current method has demonstrated to work perfectly, if the CNI can not be installed with a yaml file there is no much that KIND can do to solve that anyway.

[evankanderson]

I think I meant something like:

networking:
  disableDefaultCNI: true 
  podSubnet: 192.168.0.0/16
  CNISetup: docker.io/cillium/cillium-kind-setup
  args:
  # args from https://docs.cilium.io/en/stable/gettingstarted/k8s-install-helm/ and jayunit100's script
  - "--namespace=kube-system"
  - "--version=1.10"
  - "--nodeinit.enabled=true"
  - "--nodePort.enabled=true"
  - "--hostPort.enabled=true"
  - "--ipam.mode=kubernetes"

Which would still put the burden of shipping a CNI setup image onto the various CNI projects (which might be run as a Pod or a Job on the cluster in the kube-system namespace).

As it is, I'm sort of doing this as a post-install script for different cluster-networking types, but it would be nice to be able to have this in a single kind command.

[aojea]

that is too much cost for the project to maintain it, the reality is that if the CNI fails or doesn't work properly, they will come to open issues to KIND, not to the CNI implementation ... Another interesting point that there are projects that are moving from CNIs to use kindnetd, because it is much more reliable, simple, ....

We are exploring/suggesting/... to use KIND api so people can build these things on top of it, one example here https://github.com/aojea/kind-networking-plugins

[BenTheElder]

I don't see much difference between the proposed CNIsetup image + args and just invoking that command (docker run CNISetup args...) yourself. We've said before we're not trying to become a workflow engine.

For most CNI it would be even simpler at just kubectl apply $manifest after running kind. The problem is:

1. You can already just do that?
2. We don't have bandwidth for official support, but they already support installing this way
3. There's no standard for communicating podsubnet etc.

So we provide one lightweight out of the box solution with the portable core networking expectations per the Kubernetes spec. The issue with network policy specifically is that currently it really isn't very portable and we don't want to ship one biased version of it. Every provider also exposes additional nonstandard APIs if nothing else.

3. is also a big problem and the reason that tools that do support lots of CNI options only support specific built in manifests that have to be baked in and maintained by the project in order to use some nonstandard mechanism to integrate that information across components. For kind we want to promote using portable Kubernetes APIs and were originally only building it to test their implementation ourselves so it makes more sense to just self integrate, for a cluster with a flat topology it's pretty simple to make something relatively robust (see images/kindnetd) and very easy for us to maintain and tweak / integrate as needed.

[BenTheElder]

I think this does merit a guide similar to the current ingress situation if someone wishes to write it.

Ingress has similarly unfortunate issues where even if not for Docker's networking (no connectivity to containers without port forwarding on Mac / windows) we would have to think long and hard before shipping one due to making kind less neutral. There at least we could probably do ingress-nginx as an in-org project at least but the same did not exist for CNI so we did it ourselves.

[evankanderson]

If there's a goal of not becoming a workflow engine, I can understand that.

It would be convenient for my testing to be able to run a Job or two after the cluster is up, but it sounds like I should do that outside kind to get the cluster set up for our testing.

[aojea]

You can use this in the meantime https://github.com/aojea/kube-netpol/blob/main/README.md#install

[aojea]

Ok, this is the official one https://github.com/kubernetes-sigs/kube-network-policies

[jayunit100]

I guess this is resolved ?

Sounds like the solution suggested here is "for kind you can install a cni after the fact like calico or antrea or ovn.... or you can run the kube-network-policies daemonset", but kind itself won't bootstrap a netpol implementation for you, it's outside the scope of the project.

[aojea]

but kind itself won't bootstrap a netpol implementation for you, it's outside the scope of the project.

it is the same with the loadbalancer, you can use cloud-provider-kind or metallb, or for ingress you can choose any of them ... we try to provide the very minimum and more demanded, NetworkPolicies inside a kind cluster can only be used for testing, is not that you pretend to create a security environment for Pods inside a kind cluster 😄

[BenTheElder]

We can consider default installing this one, currently we only default install binaries/components that either:

• are used by kind itself
• are needed to pass conformance

But that line is semi-arbitrary and testing Kubernetes is important to us. Partially we want to avoid bloat (k8s installs / kind node images are already pretty large), partially to avoid "picking favorites", partially to promote portable Kubernetes APIs over e.g. kind specific behaviors/APIs wherever possible.

IF we started including network policy we would need to either bundle it with our "CNI" or add another similar knob for not-installing it.

[...] we try to provide the very minimum and more demanded, NetworkPolicies inside a kind cluster can only be used for testing, is not that you pretend to create a security environment for Pods inside a kind cluster 😄

Yes, but testing security policies is a thing even as a cluster end user, even if KIND clusters have poor security you're going to install RBAC rules etc. I think the question is:

• Do we consider these APIs to be sufficiently portable?
• If so, will we reconsider them for conformance?
• If not, we probably shouldn't bundle by default.
• How much size will this add to the cluster?

.... Re-evaluating this is on my TODO list next time we get a chance to sync :-)

[BenTheElder]

I think cloud-provider-kind is a bit different .... we'll have to think about this one as well.

[aojea]

• If so, will we reconsider them for conformance?

Conformance APIs sounds like a good bar

[aojea]

It seems feedback from users is that having network policies in kind by default is a good feature to have

/priority important-soon

Talked with @neolit123 and kubeadm policies will not accept them, so this implies we have to implement similar to the local-storage addon. So we only need to decide if network policies is opt-in or opt-out

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  disableNetworkPolicies: true

or

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
 networkPolicies: true

/assign @aojea @BenTheElder