Ephemeral ports usage metrics

[gaetanrebufet]

Hello, We recently ran into an ephemeral ports exhaustion and discovered that the linux node exporter just shows data from sockstat but no details related to:

• the ephemeral ports policy from /proc/sys/net/ipv4/ip_local_port_range
• the ephemeral ports usage (which we check manually today using netstat) Would that be a good feature candidate? Cheers.

[discordianfish]

This sound useful, whether we can include it depends on how we can gather these statistics. Exposing /proc/sys/net/ipv4/ip_local_port_range seems straight forward (though parsing needs to go into https://github.com/prometheus/procfs), finding out which ports are actually used might be tricky/more expensive. So some more research is needed first.

[SuperQ]

Figuring out ephemeral port usage is tricky, because it's not just the local port range. But it is multiplied by the 4-tuple of source and destination IP:port combinations. Just because you run out of ephemeral ports for a single destination, doesn't mean you're actually out of ephemeral ports. If change one one of the 4-tuple values and you get a whole new range of possible values.

This has come up before, and IMO really doesn't make sense as a node_exporter metric.

[discordianfish]

Good point. If we would want to support that, we'd need to support it per local-ip<->remote-ip<->remote-port tuple which we can't expose due to cardinality. But I agree that this is something you'd want to monitor. Unfortunately I can't think of a good way that would be generic enough to include in the node-exporter.. Technically we could find all open connections, group them by the tuple, then take the one that is closest to the ip_local_port_range limit, count it and expose that as `node_network_max_local_ports_used'. But I don't really like that option either.

[SuperQ]

How about making this a textfile collector tool first as a PoC?