Puppetlabs-firewall vs Kubernetes takes 10-20+ minutes to parse and add rules

[stefanlasiewski]

Describe the Bug

I'm hoping to get some attention for the bug described at https://tickets.puppetlabs.com/browse/MODULES-10983 . I've been running into it for a couple years. Another person, George Campbell, wrote the bug report MODULES-10983 a year ago, but there hasn't any movement.

In my case, I use Puppet to manage about 150 custom rules (Such as "Allow SSH from the management server to this node" or "Allow the internet to access port 443"). These nodes are part of a Kubernetes cluster. The CNI used by Kubernetes keeps it's routing rules in IPtables-- there are thousands of rules on every node.

Here's the problem:

Let's say that I have a new node, or that I delete all of the rules on my node and run puppet agent --test to re-add them. Because of the behavior seen in MODULES-10983, Puppet can take upwards of 30 minutes to write all of the rules.

It would be great to find a way to fix that.

Is https://tickets.puppetlabs.com/browse/MODULES-10983 a valid place to file bugs or should we file them here instead? If the former, can someone take a look at https://tickets.puppetlabs.com/browse/MODULES-10983?

[chelnak]

Hey! Thanks for raising this. It's certainly an interesting issue you are seeing. Having worked on this module I understand what George describes in the MODULES ticket and something I think we should investigate. I can't guarantee a quick fix but I can raise the profile of this ticket within our team.

I'll talk to @Aaron-Rob and get this bumped for community day.

With regards to tracking bugs - here is the right place. We are currently working through some steps to direct our contributors to GitHub issues in the future.

[stefanlasiewski]

Great thanks Craig!

[Antiarchitect]

Any update on this? We're suffering too.

[github-actions[bot]]

Hello! 👋

This issue has been open for a while and has had no recent activity. We've labelled it with attention-needed so that we can get a clear view of which issues need our attention.

If you are waiting on a response from us we will try and address your comments on a future Community Day.

Alternatively, if it is no longer relevant to you please close the issue with a comment.

[stefanlasiewski]

Was this rolled out? I don't see the change in https://forge.puppet.com/modules/puppetlabs/firewall/changelog , which as pf today goes up to v4.0.0.

[david22swan]

To anyone who is interested in this issue, as part of our investigation into this, the team has decided that a major rework of the module is needed. A separate issue has been opened regarding it on which discussion can take place on how we will ultimately proceed: https://github.com/puppetlabs/puppetlabs-firewall/issues/1100

[stefanlasiewski]

For the record: I had originally created anissue at https://tickets.puppetlabs.com/browse/MODULES-10983 , but that issue never gained traction. So instead, I created an issue here on GitHub.

[stek29]

dirty hack we've been using for a year -- just make the module skip rules in Kubernetes chains:

change this regex: https://github.com/puppetlabs/puppetlabs-firewall/blob/38bf1c19a1e143a302c45da2f22413a1d94e64f1/lib/puppet/provider/firewall/iptables.rb#L456

to this (adjust as needed, we're using calico, kube-proxy and cilium, so we're ignoring their chains):

unless %r{^\#\s+|^\:\S+|^COMMIT|^FATAL|^-A (KUBE-|cali-|CILIUM_)}.match?(line)

it only ignores rules in these chains, but doesn't mess with jumps to them or any other rules they might be mentioned it

[breca]

@stek29 Thanks for sharing your hack. I have a similar need using podman and generated CNI rules.

I managed to avoid having to edit this module by using ignore and it seems to do the trick for us.

[stek29]

@breca Ignore doesn't stop this module from parsing whole iptables output every time when checking/adding/removing rules we've had ignore since day 1 of using this module with Kubernetes, it still took like 10 minutes for a clean run with no changes (we have tens of thousands of rules in iptables)

[jplindquist]

We're starting to run into this too fully managing system firewalls, and the k8s manifests easily take 5-10m depending on the system. I'm was curious if there is any hope of an ignore style option for the module that can ignore or mark certain chains as unmanaged to make something like this more configurable:

dirty hack we've been using for a year -- just make the module skip rules in Kubernetes chains:

change this regex:

https://github.com/puppetlabs/puppetlabs-firewall/blob/38bf1c19a1e143a302c45da2f22413a1d94e64f1/lib/puppet/provider/firewall/iptables.rb#L456

to this (adjust as needed, we're using calico, kube-proxy and cilium, so we're ignoring their chains):

unless %r{^\#\s+|^\:\S+|^COMMIT|^FATAL|^-A (KUBE-|cali-|CILIUM_)}.match?(line)

it only ignores rules in these chains, but doesn't mess with jumps to them or any other rules they might be mentioned it

We're a little behind on the latest 7.x update, so I'm going to try and fork this for our 6.x version and try it out in the meantime...

[jplindquist]

This small change makes a huge difference in compilation and catalog times btw

Info: Loading facts
Info: Applying configuration version '1705083129'
Notice: Applied catalog in 642.12 seconds

vs.

Info: Loading facts
Info: Applying configuration version '1705087081'
Notice: Applied catalog in 10.12 seconds

Would just be nice if it was configurable as one of the options. I may try and tear this apart some more and see if I can come up with something unless someone more familiar with it has the time.