crowbar
neutron: Set nf_conntrack_{max,buckets} on compute nodes (bsc#1025691)
On compute hosts with many busy instances, we can fill the nf_conntrack table. This results in dropped packets, but can also result in instances not starting because PREROUTING rules cannot be added anymore.
The symptom is this log message from the kernel: nf_conntrack: table full, dropping packet
The fact that changes should be done to avoid this is not very well documented upstream. However, it was already met by some: http://openstack-in-production.blogspot.fr/2015/01/exceeding-tracked-connections.html
We need to increase nf_conntrack_max to avoid filling the nf_conntrack table. Right now we hardcode a higher value (1048576 instead of 65536), but later on we might want to make that configurable like openstack-ansible did: https://review.openstack.org/#/c/427716/ (although they did that because their default value is 4 times lower)
Also, when increasing nf_conntrack_max, it makes sense to increase nf_conntrack_buckets as otherwise, this results in decreased netfilter performance (linked lists to iterate upon become longer). Usually, nf_conntrack_buckets is set to nf_conntrack_max / 8.
https://bugzilla.suse.com/show_bug.cgi?id=1025691
Why is that only a problem on compute nodes? Sounds like any host that has large amount of iptable tracked flows is affected, which means any that does neutron-l3 as well. Is this dvr or non-dvr?
I think at least the modprobe conf belongs into packaging instead of crowbar. There is iirc also some systemd logic for sysctl
Why is that only a problem on compute nodes? Sounds like any host that has large amount of iptable tracked flows is affected, which means any that does neutron-l3 as well. Is this dvr or non-dvr?
It's not DVR. I was also wondering if it's needed on neutron-l3; trying to get the info for this.
I think at least the modprobe conf belongs into packaging instead of crowbar. There is iirc also some systemd logic for sysctl
If it goes in the package, then it's not settable dynamically. I know it's hardcoded right now, but it's trivial to add an attribute that will result in both the sysctl and modprobe settings to be changed. And it may be required, eventually.