How to deny all traffic to external IP and only allow some pods | Misunderstanding of NetworkSet/NetworkPolicy combination

[orimarti]

Hello,

I'd like to add a rule on a k8s cluster to deny all traffic to an external IP (let's say a redis instance out of the cluster) and only allow it from certain pods with a selector. I have tried this things and don't know if it's really possible without affecting the actual rules (Original NetworkPlicies) in the cluster.

1. If I add a GlobalNetworkPolicy denying all traffic to this external IP, then all my pods in the cluster will have an egress rule and all the egress traffic on the pods that didn't have any NetworkPolicy is denied

2. I could add another GlobalNetworkPolicy with a bigger order than the last one, allowing all traffic, then all the pods that had an egress NetworkPolicy before are going to be open to everywhere.

3. I have tried to create a GlobalNetworkPolicy with a NetworkSet but it looks like it doesn't work as I'd expect:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkSet
metadata:
  name: external-redis
  labels:
    app: networkset-redis
spec:
  nets:
  - 10.10.10.1/32
---
apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: external-redis
spec:
  selector: app == 'networkset-redis'
  types:
    - Egress
    - Ingress
  egress:
    - action: Deny
  ingress:
    - action: Deny
  order: 10

This rule doesn't deny traffic to the external redis instance, looks like the selector on the spec doesn't work although it works inside the egress and ingress rules, but then I'm in the same situation than the point 1 and 2.

Is there any solution to this? Applying rules to deny all traffic to an external IP without affecting all the Policies in the existing Pods in the cluster? Is it normal that the selector on the GlobalNetworkPolicy doesn't work for GlobalNetworkSet?

Thanks,

[coutinhop]

@orimarti I'm not sure I understand you completely, can you elaborate a little more? You have existing kubernetes NetworkPolicies already? Those are converted internally by calico with an order of 1000... So if I understand correctly, you could achieve what you want by doing 1 & 2 above, but by making your 'allow' GlobalNetworkPolicy order higher than 1000 so that it doesn't take precedence over the kubernetes NetworkPolicies (again, want to put a disclaimer that I may be misunderstanding your situation here, feel free to explain me why if that's the case).

This Calico Operator Course is also a great resource to better understand Calico NetworkPolicies (and how they work with kubernetes NetworkPolicies): https://academy.tigera.io/course/certified-calico-operator-level-1/

[orimarti]

Hi @coutinhop and thanks for your quick response!

Yes, the problem I'm facing in this cluster it's because I already have existing kubernetes NetworkPolicies. The problem with the step 2 is that when I add this allow by default GlobalNetworkPolicy, this makes that all the pods that have an egress k8s NetworkPolicy, all the traffic that was denied, now it's accepted, because it ends on this 'allow' GlobalNetworkPolicy. So every pod that had NetworkPolicies finally has all the egress traffic opened to everywhere (all the egress NetworkPolicies applied don't have any effect) My problem here is that I don't know if it's possible to deny all traffic by default to an external IP, allowing only some pods, without affecting the expected behavior in the cluster defined by the NetworkPolicies that I had before.

[coutinhop]

Hi @orimarti, sorry for the delay. I'm not sure I follow, did you get a chance to try having the ordering set up in a way that would achieve what you want? (So a global deny at order less than 1000 and a global allow with an order of more than 1000?)

Otherwise, off the top of my head, I think there is no trivial way to do what you're saying :disappointed:

[orimarti]

Hi @coutinhop , yes I tried to create a GlobalNetworkPolicy like this:

apiVersion: projectcalico.org/v3
kind: GlobalNetworkPolicy
metadata:
  name: allow-all
spec:
  types:
    - Egress
  egress:
    - action: Allow
      destination:
        nets:
        - 0.0.0.0/0
  order: 10001

The problem is that when I create a k8s NetworkPolicy, the default behaviour is to deny all not specified traffic, let's say something like this:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: test-network-policy
spec:
  podSelector:
    matchLabels:
      type: test
  policyTypes:
    - Egress
  egress:
    - to:
        - ipBlock:
            cidr: 10.10.10.1/32
      ports:
        - protocol: TCP
          port: 6379

With this NetworkPolicy the default behaviour would be to deny all traffic but 10.10.10.1:6379, but because of the GlobalNetworkPolicy allowing by default, it ends allowing all traffic on the type=test pods.

[caseydavenport]

Is there any solution to this? Applying rules to deny all traffic to an external IP without affecting all the Policies in the existing Pods in the cluster?

Fundamentally, selecting pods that previously weren't selected will impact their behavior, unfortunately.

Typical models for this sort of thing with what's currently supported:

• Default-deny all traffic, and then use allow-list rules to allow a subset of pods egress access to the external CIDR.
• Default-allow all traffic, and use deny-list rules to deny a subset of pods access to the external CIDR.

We are likely going to enhance Calico policies to provide an option to disable that default-deny-on-selection behavior, though, which might make this easier. This is something we've got on our roadmap at the moment. With that, you could do something like this:

• GNP rule that denies access outside of the cluster, but doesn't impact default-deny of selected pods.
• NetworkPolicy that allows a subset of pods access to the external IP.

Is it normal that the selector on the GlobalNetworkPolicy doesn't work for GlobalNetworkSet?

I would expect GNP selectors to select GNS.