network-policy-api
network-policy-api copied to clipboard
Published
20 hours ago •
kubernetes-sigs
kubernetes-sigs
feat(Policy Assistant): support LoadBalancer and NodePort services in "generate" conformance tests
Overview
Today in Policy Assistant and Cyclonus, the "generate" conformance tests only test k8s services with type=ClusterIP.
This PR adds 18 test cases in total for traffic to LoadBalancer and NodePort services (for both externalTrafficPolicy values).
Test Cases
For each service type/externalTrafficPolicy, testing 1) default deny ingress and 2) allowing ingress (via Pods and node CIDR if needed). Similar for egress.
LoadBalancer Test Cases
Tests send traffic from each Pod to each LoadBalancer service's external IP.
The source Pod's node is expected to "intercept" this traffic and redirect to the one backend Pod (which may be on another node).
In the case of externalTrafficPolicy =Local, the backend Pod must be on the source node. Therefore, tests ignore inter-node probes.
In the case of externalTrafficPolicy =Cluster, if the backend Pod is on another node, the CNI will SNAT traffic to the node IP. For some NetworkPolicy implementations, the source Pod info is lost in this case; to allow traffic requires a policy that allows traffic from the source node.
NodePort Test Cases
Tests send traffic from each Pod to each NodePort service's node port.
In the case of externalTrafficPolicy =Local, tests send traffic to the destination Pod's node.
In the case of externalTrafficPolicy =Cluster, tests send traffic to the source Pod's node (since this externalTrafficPolicy supports redirecting traffic to another node).
Verifying this change
New Test Cases
Verified traffic goes to correct services and that tests pass for two NetworkPolicy implementations.
There are 18 test cases. For Azure NPM, 2 must be ignored (see command below). For Cilium, 3 cases should be ignored since Cilium doesn't support ipBlock by default.
Old Test Cases
Verified that nothing seems to break. Only ClusterIP services are created.
Example runs for two NetworkPolicy implementations
Cilium:
policy-assistant generate --pod-creation-timeout-seconds 600 --server-protocol TCP,UDP --ignore-loopback --include special-services --exclude ip-block-no-except
Azure NPM:
policy-assistant generate --pod-creation-timeout-seconds 600 --server-protocol TCP,UDP --ignore-loopback --include special-services --exclude cni-brings-source-pod-info-to-other-node
CLI Outputs
All 18 test cases (none are ignored here):
test #1: LoadBalancer with externalTrafficPolicy=Cluster: deny all ingress
- tags: deny-all, direction, external-traffic-policy-cluster, ingress, loadbalancer, rule, special-services
test #2: LoadBalancer with externalTrafficPolicy=Cluster: allow ingress from pods
- tags: all-namespaces, cni-brings-source-pod-info-to-other-node, direction, external-traffic-policy-cluster, ingress, loadbalancer, peer-pods, special-services
test #3: LoadBalancer with externalTrafficPolicy=Cluster: allow ingress from pods and nodes
- tags: all-namespaces, direction, external-traffic-policy-cluster, ingress, ip-block-no-except, loadbalancer, no-cni-source-pod-info-to-other-node, peer-ipblock, peer-pods, special-services
test #4: LoadBalancer with externalTrafficPolicy=Cluster: deny all egress
- tags: deny-all, direction, egress, external-traffic-policy-cluster, loadbalancer, rule, special-services
test #5: LoadBalancer with externalTrafficPolicy=Cluster: allow egress to pods
- tags: all-namespaces, direction, egress, external-traffic-policy-cluster, loadbalancer, peer-pods, special-services
test #6: NodePort with externalTrafficPolicy=Cluster: deny all ingress (to source pod's node)
- tags: deny-all, direction, external-traffic-policy-cluster, ingress, nodeport, rule, special-services, to-source-pod-node
test #7: NodePort with externalTrafficPolicy=Cluster: allow ingress from pods (to source pod's node)
- tags: all-namespaces, cni-brings-source-pod-info-to-other-node, direction, external-traffic-policy-cluster, ingress, nodeport, peer-pods, special-services, to-source-pod-node
test #8: NodePort with externalTrafficPolicy=Cluster: allow ingress from pods and nodes (to source pod's node)
- tags: all-namespaces, direction, external-traffic-policy-cluster, ingress, ip-block-no-except, no-cni-source-pod-info-to-other-node, nodeport, peer-ipblock, peer-pods, special-services, to-source-pod-node
test #9: NodePort with externalTrafficPolicy=Cluster: deny all egress (to source pod's node)
- tags: deny-all, direction, egress, external-traffic-policy-cluster, nodeport, rule, special-services, to-source-pod-node
test #10: NodePort with externalTrafficPolicy=Cluster: allow egress to pods (to source pod's node)
- tags: all-namespaces, direction, egress, external-traffic-policy-cluster, nodeport, peer-pods, special-services, to-source-pod-node
test #11: LoadBalancer with externalTrafficPolicy=Local: deny all ingress
- tags: deny-all, direction, external-traffic-policy-local, ingress, loadbalancer, rule, special-services
test #12: LoadBalancer with externalTrafficPolicy=Local: allow ingress from pods
- tags: all-namespaces, direction, external-traffic-policy-local, ingress, loadbalancer, peer-pods, special-services
test #13: LoadBalancer with externalTrafficPolicy=Local: deny all egress
- tags: deny-all, direction, egress, external-traffic-policy-local, loadbalancer, rule, special-services
test #14: LoadBalancer with externalTrafficPolicy=Local: allow egress to pods
- tags: all-namespaces, direction, egress, external-traffic-policy-local, loadbalancer, peer-pods, special-services
test #15: NodePort with externalTrafficPolicy=Local: deny all ingress (to destination pod's node)
- tags: deny-all, direction, external-traffic-policy-local, ingress, nodeport, rule, special-services, to-destination-pod-node
test #16: NodePort with externalTrafficPolicy=Local: allow ingress from pods (to destination pod's node)
- tags: all-namespaces, direction, external-traffic-policy-local, ingress, nodeport, peer-pods, special-services, to-destination-pod-node
test #17: NodePort with externalTrafficPolicy=Local: deny all egress (to destination pod's node)
- tags: deny-all, direction, egress, external-traffic-policy-local, nodeport, rule, special-services, to-destination-pod-node
test #18: NodePort with externalTrafficPolicy=Local: allow egress to pods and nodes (to destination pod's node)
- tags: all-namespaces, direction, egress, external-traffic-policy-local, ip-block-no-except, nodeport, peer-ipblock, peer-pods, special-services, to-destination-pod-node
Result of Tests
Example run using the Azure NPM cluster:
$ policy-assistant generate --pod-creation-timeout-seconds 600 --server-protocol TCP,UDP --ignore-loopback --include special-services --exclude cni-brings-source-pod-info-to-other-node
...
finished policy #16
SummaryTable:
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| TEST | RESULT | STEP/TRY | WRONG | RIGHT | IGNORED | TCP | SCTP | UDP |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 1: LoadBalancer with | passed | | | | | | | |
| externalTrafficPolicy=Cluster: | | | | | | | | |
| deny all ingress | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 2: LoadBalancer with | passed | | | | | | | |
| externalTrafficPolicy=Cluster: | | | | | | | | |
| allow ingress from pods and | | | | | | | | |
| nodes | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 3: LoadBalancer with | passed | | | | | | | |
| externalTrafficPolicy=Cluster: | | | | | | | | |
| deny all egress | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 4: LoadBalancer with | passed | | | | | | | |
| externalTrafficPolicy=Cluster: | | | | | | | | |
| allow egress to pods | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 5: NodePort with | passed | | | | | | | |
| externalTrafficPolicy=Cluster: | | | | | | | | |
| deny all ingress (to source | | | | | | | | |
| pod's node) | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 6: NodePort with | passed | | | | | | | |
| externalTrafficPolicy=Cluster: | | | | | | | | |
| allow ingress from pods and | | | | | | | | |
| nodes (to source pod's node) | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 7: NodePort with | passed | | | | | | | |
| externalTrafficPolicy=Cluster: | | | | | | | | |
| deny all egress (to source | | | | | | | | |
| pod's node) | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 8: NodePort with | passed | | | | | | | |
| externalTrafficPolicy=Cluster: | | | | | | | | |
| allow egress to pods (to | | | | | | | | |
| source pod's node) | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 9: LoadBalancer with | passed | | | | | | | |
| externalTrafficPolicy=Local: | | | | | | | | |
| deny all ingress | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 32 | 49 | 64 / 64 (100%) | - | 64 / 64 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 10: LoadBalancer with | passed | | | | | | | |
| externalTrafficPolicy=Local: | | | | | | | | |
| allow ingress from pods | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 32 | 49 | 64 / 64 (100%) | - | 64 / 64 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 11: LoadBalancer with | passed | | | | | | | |
| externalTrafficPolicy=Local: | | | | | | | | |
| deny all egress | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 32 | 49 | 64 / 64 (100%) | - | 64 / 64 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 12: LoadBalancer with | passed | | | | | | | |
| externalTrafficPolicy=Local: | | | | | | | | |
| allow egress to pods | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 32 | 49 | 64 / 64 (100%) | - | 64 / 64 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 13: NodePort with | passed | | | | | | | |
| externalTrafficPolicy=Local: | | | | | | | | |
| deny all ingress (to | | | | | | | | |
| destination pod's node) | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 14: NodePort with | passed | | | | | | | |
| externalTrafficPolicy=Local: | | | | | | | | |
| allow ingress from pods (to | | | | | | | | |
| destination pod's node) | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 15: NodePort with | passed | | | | | | | |
| externalTrafficPolicy=Local: | | | | | | | | |
| deny all egress (to | | | | | | | | |
| destination pod's node) | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| 16: NodePort with | passed | | | | | | | |
| externalTrafficPolicy=Local: | | | | | | | | |
| allow egress to pods and nodes | | | | | | | | |
| (to destination pod's node) | | | | | | | | |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
| | | Step 1, try 1 | 0 | 72 | 9 | 144 / 144 (100%) | - | 144 / 144 (100%) |
+--------------------------------+--------+---------------+-------+-------+---------+------------------+------+------------------+
rule counts:
+----------+--------+--------+----------+
| FEATURE | PASSED | FAILED | PASSED % |
+----------+--------+--------+----------+
| deny-all | 8 | 0 | 100 |
+----------+--------+--------+----------+
special-services counts:
+--------------------------------------+--------+--------+----------+
| FEATURE | PASSED | FAILED | PASSED % |
+--------------------------------------+--------+--------+----------+
| to-source-pod-node | 4 | 0 | 100 |
| nodeport | 8 | 0 | 100 |
| external-traffic-policy-local | 8 | 0 | 100 |
| to-destination-pod-node | 4 | 0 | 100 |
| external-traffic-policy-cluster | 8 | 0 | 100 |
| loadbalancer | 8 | 0 | 100 |
| no-cni-source-pod-info-to-other-node | 2 | 0 | 100 |
+--------------------------------------+--------+--------+----------+
direction counts:
+---------+--------+--------+----------+
| FEATURE | PASSED | FAILED | PASSED % |
+---------+--------+--------+----------+
| ingress | 8 | 0 | 100 |
| egress | 8 | 0 | 100 |
+---------+--------+--------+----------+
peer-ipblock counts:
+--------------------+--------+--------+----------+
| FEATURE | PASSED | FAILED | PASSED % |
+--------------------+--------+--------+----------+
| ip-block-no-except | 3 | 0 | 100 |
+--------------------+--------+--------+----------+
peer-pods counts:
+----------------+--------+--------+----------+
| FEATURE | PASSED | FAILED | PASSED % |
+----------------+--------+--------+----------+
| all-namespaces | 8 | 0 | 100 |
+----------------+--------+--------+----------+
Pass/Fail for probes on protocols:
+---------------+--------+--------+----------+
| PROTOCOL | PASSED | FAILED | PASSED % |
+---------------+--------+--------+----------+
| probe on TCP | 1984 | 0 | 100 |
| probe on SCTP | 0 | 0 | 0 |
| probe on UDP | 1984 | 0 | 100 |
+---------------+--------+--------+----------+
Feature results:
| Tag | Result |
| --- | --- |
| action | 16 / 16 = 100% ✅ |
| - action: create policy | 16 / 16 = 100% ✅ |
| egress | 16 / 16 = 100% ✅ |
| - 0 port/protocols | 4 / 4 = 100% ✅ |
| - 0 rules | 12 / 12 = 100% ✅ |
| - 1 peer | 3 / 3 = 100% ✅ |
| - 1 rule | 4 / 4 = 100% ✅ |
| - 2+ peers | 1 / 1 = 100% ✅ |
| - IPBlock (no except) | 1 / 1 = 100% ✅ |
| - peer namespace selector empty | 4 / 4 = 100% ✅ |
| - peer pod selector nil | 4 / 4 = 100% ✅ |
| general | 16 / 16 = 100% ✅ |
| - policy with egress | 4 / 4 = 100% ✅ |
| - policy with ingress | 4 / 4 = 100% ✅ |
| - target: empty pod selector | 16 / 16 = 100% ✅ |
| - target: specific namespace | 16 / 16 = 100% ✅ |
| ingress | 16 / 16 = 100% ✅ |
| - 0 port/protocols | 4 / 4 = 100% ✅ |
| - 0 rules | 12 / 12 = 100% ✅ |
| - 1 peer | 2 / 2 = 100% ✅ |
| - 1 rule | 4 / 4 = 100% ✅ |
| - 2+ peers | 2 / 2 = 100% ✅ |
| - IPBlock (no except) | 2 / 2 = 100% ✅ |
| - peer namespace selector empty | 4 / 4 = 100% ✅ |
| - peer pod selector nil | 4 / 4 = 100% ✅ |
Tag results:
| Tag | Result |
| --- | --- |
| direction | 16 / 16 = 100% ✅ |
| - egress | 8 / 8 = 100% ✅ |
| - ingress | 8 / 8 = 100% ✅ |
| peer-ipblock | 3 / 3 = 100% ✅ |
| - ip-block-no-except | 3 / 3 = 100% ✅ |
| peer-pods | 8 / 8 = 100% ✅ |
| - all-namespaces | 8 / 8 = 100% ✅ |
| rule | 8 / 8 = 100% ✅ |
| - deny-all | 8 / 8 = 100% ✅ |
| special-services | 16 / 16 = 100% ✅ |
| - external-traffic-policy-cluster | 8 / 8 = 100% ✅ |
| - external-traffic-policy-local | 8 / 8 = 100% ✅ |
| - loadbalancer | 8 / 8 = 100% ✅ |
| - no-cni-source-pod-info-to-other-node | 2 / 2 = 100% ✅ |
| - nodeport | 8 / 8 = 100% ✅ |
| - to-destination-pod-node | 4 / 4 = 100% ✅ |
| - to-source-pod-node | 4 / 4 = 100% ✅ |
[APPROVALNOTIFIER] This PR is APPROVED
This pull-request has been approved by: huntergregory
The full list of commands accepted by this bot can be found here.
The pull request process is described here
Needs approval from an approver in each of these files:
- ~~cmd/policy-assistant/OWNERS~~ [huntergregory]
Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment
Deploy Preview for kubernetes-sigs-network-policy-api ready!
| Name | Link |
|---|---|
| Latest commit | 0d0e470332d4b88854cfacdd931c19c0b2dce89a |
| Latest deploy log | https://app.netlify.com/sites/kubernetes-sigs-network-policy-api/deploys/67ad1b567752a8000855f743 |
| Deploy Preview | https://deploy-preview-287--kubernetes-sigs-network-policy-api.netlify.app |
| Preview on mobile | Toggle QR Code...Use your smartphone camera to open QR code link. |
To edit notification comments on pull requests, go to your Netlify site configuration.
![netlify[bot] avatar](https://avatars.githubusercontent.com/in/13473?v=4 "Published by a pub.dev verified publisher"){kind=small}
The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the PR is closed
You can:
- Mark this PR as fresh with
/remove-lifecycle stale - Close this PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all PRs.
This bot triages PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the PR is closed
You can:
- Mark this PR as fresh with
/remove-lifecycle rotten - Close this PR with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle rotten
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the PR is closed
You can:
- Reopen this PR with
/reopen - Mark this PR as fresh with
/remove-lifecycle rotten - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/close
@k8s-triage-robot: Closed this PR.
In response to this:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.
This bot triages PRs according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied- After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied- After 30d of inactivity since
lifecycle/rottenwas applied, the PR is closedYou can:
- Reopen this PR with
/reopen- Mark this PR as fresh with
/remove-lifecycle rotten- Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.