aws
NetworkPolicy that results in multiple PolicyEndpoint resources has unexpected periods of "Default Deny enabled"
When a NetworkPolicy resource results in multiple PolicyEndpoint resources due to a large number of endpoints (e.g. when they have been chunked) we have observed log messages from https://github.com/aws/aws-network-policy-agent during pod startup with the following content: Default Deny enabled on Egress (from https://github.com/aws/aws-network-policy-agent/blob/17489e56193ee53793831e7f07346ddd15173a6a/controllers/policyendpoints_controller.go#L422).
This is unexpected as the NetworkPolicy resources seeing this issue define both egress & ingress in the spec and have both Ingress & Egress listed in policyTypes so no default deny should take place. Additionally this occurs when NETWORK_POLICY_ENFORCING_MODE=standard so we don't expect blocking like this to occur during startup for a NetworkPolicy that contains the appropriate spec & policyTypes.
What we believe to be happening is that chunking is resulting in PolicyEndpoint resources that all have podIsolation set to both Ingress & Egress values (which reflects the underlying policy) but in some of these PolicyEndpoint resources there will only be ingress rules listed in the spec resulting in the egress rules count being equal to 0 and having this default deny logic applying for some period of time until a later PolicyEndpoint is applied that has the egress rules to apply.
To better demonstrate this I've included some sample resources below which I've redacted and trimmed to focus on the relevant parts. I can provide additional logs and/or configuration privately if required but only at info level for now as that is what we currently log at to avoid policy decision logs at ACCEPT level being logged due to https://github.com/aws/aws-network-policy-agent/issues/467. I've also only captured the contents of /var/log/aws-routed-eni as the log script from https://github.com/aws/amazon-vpc-cni-k8s/blob/master/docs/troubleshooting.md#collecting-node-level-tech-support-bundle-for-offline-troubleshooting does not currently work on Bottlerocket which we run.
NetworkPolicy
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
...
spec:
egress:
- to:
- ipBlock:
cidr: 0.0.0.0/0
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: example
podSelector:
matchLabels:
app.kubernetes.io/name: prometheus
ports:
- port: 4000
protocol: TCP
- from:
- ipBlock:
cidr: removed
- ipBlock:
cidr: removed
- ipBlock:
cidr: removed
ports:
- port: 3000
protocol: TCP
podSelector:
matchLabels:
app.kubernetes.io/name: example
policyTypes:
- Ingress
- Egress
PolicyEndpoints
(1) - the one missing egress but has Egress listed in podIsolation
apiVersion: networking.k8s.aws/v1alpha1
kind: PolicyEndpoint
...
spec:
ingress:
- cidr: removed
ports:
- port: 4000
protocol: TCP
podIsolation:
- Ingress
- Egress
podSelector:
matchLabels:
app.kubernetes.io/name: example
podSelectorEndpoints:
- removed (there's a lot)
(2)
apiVersion: networking.k8s.aws/v1alpha1
kind: PolicyEndpoint
...
spec:
egress:
- cidr: 0.0.0.0/0
ingress:
- cidr: removed
ports:
- port: 3000
protocol: TCP
- cidr: removed
ports:
- port: 3000
protocol: TCP
- cidr: removed
ports:
- port: 3000
protocol: TCP
podIsolation:
- Ingress
- Egress
podSelector:
matchLabels:
app.kubernetes.io/name: example
podSelectorEndpoints:
- removed (there's a lot)
@essh Hey, thank you for bringing this up! Can you confirm whether you observed actual egress traffic being blocked during this window, or is the concern primarily the misleading log message? This will help determine if we need to fix just the logging or if there's actual enforcement happening despite standard mode.
@essh Hey, thank you for bringing this up! Can you confirm whether you observed actual egress traffic being blocked during this window, or is the concern primarily the misleading log message? This will help determine if we need to fix just the logging or if there's actual enforcement happening despite standard mode.
Based on the policy decision logs we collect we are seeing quite a number of unexpected DENY verdict around pod startup on a larger cluster that has chunked the PolicyEndpoint that connects back to policies where this happened. We aren't seeing this on other clusters that are smaller and not logging this message so I believe this is having some unexpected blocking impact but I have not really dug too far into this.
@essh what is the scale of the cluster where you are applying network policies? can you also share some more details for pods which are seeing these DENY like how many Network Policies are they targeted from and how much is the chunks for each Network Policy?
I believe this issue will show up more as scale increases, this is somewhat similar to https://github.com/aws/amazon-network-policy-controller-k8s/issues/179
For example - you have a pod which has NETWORK_POLICY_ENFORCING_MODE=standard and that pod is targeted by 10 PolicyEndpoints (could be from same Network Policy of different). Now this controller is creating those PolicyEndpoint chunks which are watched by network-policy-agent. Agent gets the reconciliation request and sees subset of the PolicyEndpoints (as other PE can still be processing by NPC or belong to separate NetworkPolicy which are yet to reconcile) and moves pod state from Default_Allow to Policies_Applied (although partial rules are applied yet).
The scale is reasonable but not what I'd consider excessive, we see up to 6000 pods within the largest cluster across up to ~450 nodes.
The largest policy endpoints would target anything from 1000+ pods (some close to all pods as above) which is generally split across 2-5 PolicyEndpoint chunks.
In terms of how many policies they are targeted from, a podSelector will normally be quite specific about what it targets so there won't be much overlap there if any. Most of the rules are on the ingress from side and at most there would generally only be about 3-4 of these targeting based on namespaceSelector and/or podSelector with a few more ipBlock with cidr. However, some of these do include selectors that would cover thousands of pods as above.