prometheus-nats-exporter icon indicating copy to clipboard operation
prometheus-nats-exporter copied to clipboard

Export connection info

Open rigtorp opened this issue 6 years ago • 3 comments

What: Export information about individual connections

Why: To allow prometheus alerting on slow clients and track bandwidth usage by client.

How: Something like this:

gnatsd_connz_out_bytes{name="xxx", ip="host", port="port", cid="yyy"}
gnatsd_connz_in_bytes{name="xxx", ip="host", port="port", cid="yyy"}
... etc

SNMP exporter uses similar pattern:

ifHCOutOctets{ifAlias="",ifDescr="Unit: 1 Slot: 0 Port: 27 10G - Level",ifIndex="27",ifName="Te1/0/27"} 48839
ifInErrors{ifAlias="",ifDescr="Unit: 1 Slot: 0 Port: 23 10G - Level",ifIndex="23",ifName="Te1/0/23"} 0

rigtorp avatar May 28 '19 18:05 rigtorp

We need this feature too to be able to define limits and alerts based on number of connections and subscriptions per client. Furthermore showing additional connection option like MaxPubInFlight would help to prevent "rogue" clients from crashing the whole cluster.

wutkemtt avatar Feb 17 '20 09:02 wutkemtt

I dont think this will work well, the way prom stores data this will lead to a huge explosion of data even on relatively small networks

ripienaar avatar Feb 17 '20 09:02 ripienaar

Here are guidelines from the author/maintainers:

As a general rule of thumb I'd avoid having any metric whose cardinality on a /metrics could potentially go over 10 due to growth in label values.

https://www.robustperception.io/cardinality-is-key

Your proposal even if 1 client connects 10 times from the same host to the server we'd already be on 40 different metrics (port unique each time). Even just including the client name would quickly become a performance issue on the prom server.

ripienaar avatar Feb 17 '20 09:02 ripienaar