Network Policy

[plumdog]

I would like to be able to use network policies on pods that have iam.amazonaws.com/role: ... on them.

I imagined this would mean adding a network policy that looks like:

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  ...
spec:
  egress:
  - ports:
    - port: 80
      protocol: TCP
    to:
    - ipBlock:
        cidr: 169.254.169.254/32
  podSelector:
    matchLabels:
      app: myapp
  policyTypes:
  - Egress

However, this still blocks a curl http://169.254.169.254 from the pod.

A policy that does work looks like:

apiVersion: extensions/v1beta1
kind: NetworkPolicy
metadata:
  ...
spec:
  egress:
  - ports:
    - port: 8181
      protocol: TCP
    to:
    - ipBlock:
        cidr: [my cluster host instances IP range]
    - ipBlock:
        cidr: [my kops nonMasqueradeCIDR range]
  podSelector:
    matchLabels:
      app: myapp
  policyTypes:
  - Egress

That is: I have to specify the host instances IP range, and the port that the kiam agent is listening on, 8181. I think this means that the network policies are applied after kiam does its iptables magic.

Is my network policy above the best that I can do? Is there a better way to go about solving this problem?

[pingles]

If you're trying to make sure Pods can't talk to the metadata API you don't need to- when you deploy the Kiam agent as a DaemonSet on the nodes running user workloads (and you use the --iptables flag) it'll create a rule to intercept requests to http://169.254.169.254; Pods won't be able to talk to the metadata API directly.

Please let me know if I've misunderstood?

[plumdog]

My use-case is actually the other way around - I have application pods that have network policies selecting them, and allowing some egress traffic (namely "this application pod may talk to those application pods"). Once I have a network policy selecting the pod, anything not explicitly allowed is denied (I'm pretty sure that is correct).

So once I'm selecting my application pod with a network policy at all, I need a policy to say "please let this pod talk to the kiam metadata service" if I want it to be able to do so, otherwise it will be blocked.

[pingles]

Ah ok, thanks, I understand now.

Unfortunately I'm not sure I can add anything to help at the moment. I'll leave this issue open in case others have good ideas on what we could do.

[vdoodala]

Is there any update for this issue, even I am stuck with the same issue when applying network policies.

[blairboy362]

I'd also be interested to know if anyone has come up with a decent solution to this.

In our case we tried following Istio's recommendations of network policies to prevent arbitrary egress from an EKS-based cluster. Due to the nature of the AWS VPC CNI we're unable to do what the OP did as the subnets of the nodes overlaps with the subnets of the pods (adding an egress for the whole VPC subnet on 8181 was deemed unacceptable for reasons that don't matter here). We're currently trying to work around it using istio but it's far from ideal.

[discreet]

I'm also struggling with this. My network policies are allowing ingress/egress to 10.0.0.0/8 which don't work and still don't work when I add 169.254.169.254/32. I don't have any port restrictions or anything else. Just those two things for ingress and egress.

[mathewmoon]

This may help: https://www.bluematador.com/blog/iam-access-in-kubernetes-kube2iam-vs-kiam

An additional feature that is not present in kube2iam is restricting EC2 metadata api access. By default, kiam denies access to other metadata api paths that can allow pods to discover information about the node they are running on. It is common for monitoring tools in particular to use the metadata API to associate your Kubernetes resources with AWS resources. This can be configured via the --whitelist-route-regexp flag. To allow all routes use .* as the value.

I banged my head against this wall when we set up kiam.

[geekofalltrades]

I came across this while evaluating replacing kube2iam with kiam and think I can help.

Currently, in our kube2iam deployment, we have 169.254.169.254:80 DNAT'd to 127.0.0.1:8181 on each node, where kube2iam is listening.

For pods with Network Policies, we have added this as an egress rule:

spec:
  egress:
    - to:
        - ipBlock:
            cidr: 127.0.0.1/32
      ports:
        - protocol: TCP
          port: 8181

This allows the pod to reach the metadata service at http://169.254.169.254. We're using calico networking, although I don't think that should make a difference.

Presumably under kiam, this would allow the pod to reach the kiam agent, which is taking the place of kube2iam in our setup.

UPDATE: We switched to kiam, and this networkpolicy is working in our clusters.

[jaygorrell]

We're trying to make this work by just adding a NetworkPolicy that allows connections to the name: kube-system namespace (yes it has that label) but that doesn't seem to be working either. Is there something with hostNetwork that prevents namespace selectors from working?

[ipochi]

@jaygorrell if the pod is running with hostNetwork: true Kubernetes NetworkPolicy or Calico's NetworkPolicy is not applied.

Calico's GlobalNetworkPolicy works but you cannot use selector, as ingress/egress traffic from host networked pod is not indistinguishable from other traffic from the host.