flannel FORWARD chain rules are placed after terminating user rules

[PAI5REECHO]

The issue I have is that flannel inserts its iptables rules after some terminating user rules I've made for logging purposes.

Expected Behavior

For flannel to provide some way to change the insertion order of its iptables rules in the FORWARD chain.

Current Behavior

# iptables -nvL FORWARD --line-numbers
Chain FORWARD (policy DROP 0 packets, 0 bytes)
num   pkts bytes target                  prot opt in     out     source               destination
1    40659 2294K KUBE-FORWARD            all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* kubernetes forwarding rules */
2    40659 2294K KUBE-SERVICES           all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes service portals */
3    40295 2260K KUBE-EXTERNAL-SERVICES  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate NEW /* kubernetes externally-visible service portals */
4    40876 2293K LOG                     all  --  *      *       0.0.0.0/0            0.0.0.0/0            LOG flags 8 level 4 prefix "FORWARD (rejecting): "
5    40876 2293K REJECT                  all  --  *      *       0.0.0.0/0            0.0.0.0/0            /* Log all rejected packets */ reject-with icmp-host-prohibited
6        0     0 ACCEPT                  all  --  *      *       10.1.0.0/16          0.0.0.0/0            /* flanneld forward */
7        0     0 ACCEPT                  all  --  *      *       0.0.0.0/0            10.1.0.0/16          /* flanneld forward */

Steps to Reproduce (for bugs)

1. Set a default DROP policy for all filter chains, and also add additional logging and rejection rules for the FORWARD chain.
2. flannel inserts it's rules AFTER the REJECT rules so they are never reached in the FORWARD chain

Your Environment

• Flannel version: v0.18.0
• Backend used (e.g. vxlan or udp): I don't know how to check this
• Etcd version: 3.3.27
• Kubernetes version (if used): v1.23.8
• Operating System and version: Linux 5.18.10, NixOS, 22.11 (Raccoon), 22.11.20220619.0d68d7c

[rbrtbnfgl]

Flannel append its rule right after the all the iptables rule to avoid conflicts with user firewall configured rules. It will not be easy to manages the ordering of the rules because you have to put flannel rules in between your REJECT rules and kubernetes rules.

[deric]

@rbrtbnfgl Would it make sense for flannel to append rules e.g. to KUBE-FORWARD chain? Sorry for dumb question, I'm just trying to understand how to avoid conflicts with existing iptables rules.

[rbrtbnfgl]

I think it will relate to much flannel with the kubernetes implementation. Maybe create a new chain could be a possible solution.

[PAI5REECHO]

Linking related issues #1033 & #1542 in regards to iptables rule ordering

[deric]

Instead of jumping FORWARD -> KUBE-FORWARD -> FORWARD -> DROP

-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
# default forward DROP
-A FORWARD -j DROP
# unreachable rules in FORWARD chain
-A FORWARD -s 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
# end of unreachable rules
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT

we could use less obtrusive way FORWARD -> KUBE-FORWARD -> ACCEPT

-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
# default forward DROP
-A FORWARD -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -s 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT
-A KUBE-FORWARD -d 10.244.0.0/16 -m comment --comment "flanneld forward" -j ACCEPT

of course this makes sense only if Kubernetes is used.

Another solution would be disabling managing FORWARD rules (as implemented in #978) and manage such rules externally.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.