netlink icon indicating copy to clipboard operation
netlink copied to clipboard

netlink: add ExecuteFunc method to process large responses with low mem overhead

Open bobby-stripe opened this issue 8 months ago • 0 comments

One of the great things you can do with netlink is query for detailed stats on all open TCP connections, including e.g. retransmits, how much data is in socket buffers, the computed BBR bandwidth estimates, etc. Unfortunately, if you ask the kernel for this info on a host with a lot of connections, the []netlink.Message slice returned by netlink/Conn.Execute can take 50+ MB of space, when the slice will be (likely) be thrown away after its iterated over once.

This PR adds a new method ExecuteFunc where a user can pass in a callback function that is called once per message. Becuase of how Conn.receive/the netlink recvmsg API works in batches/streaming, this dramatically reduces the peak RSS/memory overhead required to process a large stream of netlink message responses.

To try to minimize duplication, I refactored receive into receiveEach which operates on a callback, changed lockedReceive to call receiveEach, and then added a new lockedReceiveEach which is callled by ExecuteFunc (I started with the Each suffix, but then saw that a bunch of the callback variants in the Go stdlib have a Func suffix. LMK if you prefer one or the other or something else and I can standardize the naming).

bobby-stripe avatar Jun 24 '24 21:06 bobby-stripe