gnmic icon indicating copy to clipboard operation
gnmic copied to clipboard

Add gNMI Extension field parsing support

Open Ichabond opened this issue 6 months ago • 5 comments

gNMI allows for the use of an extension field in each top-level message of the gNMI RPCs: https://github.com/openconfig/reference/blob/master/rpc/gnmi/gnmi-extensions.md Given this is an arbitrary Protobyte payload, the default gNMI protobufs can't decode the payload contained within the field. This PR adds the necessary configuration options to load in an arbitrary Protobuf file per extension identifier, with a message-name to lookup the message type.

In this change:

  1. A new function DecodeExtension is added. This function uses protoreflect to dynamically marshal arbitrary protoBytes into JSON. The loaded JSON is then put back in the Extension message as bytes (mainting type)
  2. The Target type has an ExtensionProtoMap added, allowing for the lookup of Extension IDs to loaded-in protobufs
  3. The required changes to the TargetConfig type to support loading in the new configuration
  4. Modified collector.go to output the gNMI message after inlining the decoded protoBytes
  5. Loading in the protobufs was added to app/target.go: parseExtensionProtos. This uses the Parser provided by protoreflect/desc/protoparse
  6. Added functionality to event.go to insert the K/Vs provided by the Extension as Tags. Given we come from JSON, all numbers are float64, so the only 2 types supported currently are string and float64
  7. Minor helper function to turn the arbitrary JSON into an arbitrary map.

This has been tested with a device emiting an extension field:

[gnmic] target "edge01_test01": gNMI Subscribe Response: &{
SubscriptionName:port_stats
SubscriptionConfig:{"name":"port_stats","paths":["/interfaces/interface/"],"mode":"STREAM","stream-mode":"SAMPLE","encoding":"JSON","sample-interval":15000000000,"heartbeat-interval":15000000000,"outputs":["prom-scrape-output"]}
Response:
update:{timestamp:1723653363502452302  prefix:{elem:{name:"interfaces"}  elem:{name:"interface"  key:{key:"name"  value:"et-1/0/3"}}}
update:{path:{elem:{name:"state"}  elem:{name:"hardware-port"}}  val:{json_val:"\"FPC1:PIC0:PORT3\""}}
update:{path:{elem:{name:"state"}  elem:{name:"transceiver"}}  val:{json_val:"\"FPC1:PIC0:PORT3:Xcvr0\""}}}
extension:{registered_ext:{id:1
    msg:"{\"systemId\":\"edge01_test01\",\"componentId\":65535,\"sensorName\":\"sensor_1005_2_1\",\"subscribedPath\":\"/interfaces/interface/\",\"streamedPath\":\"/interfaces/interface/\",\"component\":\"chassisd\",\"sequenceNumber\":\"770002\",\"payloadGetTimestamp\":\"1723653363502\",\"streamCreationTimestamp\":\"1723653361858\",\"exportTimestamp\":\"1723653363504\",\"streamId\":\"PERIODIC\"}"}}}

Which is then properly rendered to a Prometheus metric:

gnmi_interfaces_interface_state_hardware_port{component="chassisd",componentId="65535",hardware_port="FPC1:PIC0:PORT3",interface_name="et-1/0/3",metric_source="edge01_test01",subscription_name="port_stats",systemId="edge01_test01"} 1

Note that some label-drop rules have been added to remove the spurious labels to avoid a cardinality explosion.

Ichabond avatar Aug 14 '24 16:08 Ichabond