Model prediction events and streaming pipelines

[ssheng]

Goals

• Instrument input and output of every prediction in both api_server and runners
• Schematized
• Agnostic of the wire protocols, e.g. HTTP, gRPC
• Transferred in binary formats
• Designed to be shipped off the box to remote data wire houses

Design Considerations

• Data container will be responsible for encoding the input and output
• The msgpack format is a good encoding candidate
• Event is sent directly to a forwarding agent like Fluent Bit
• Create an abstract interface for the event exporter
  • By default, the event exporter is no-op
  • The event exporter can be configured to ship events to the forwarding agent or write events to files

Schema

{
  "component": "iris_runner",
  "batched": "False",
  "inputs": [
    [5, 5, 5, 5,],
  ],
  "outputs": [
    2,
  ],
  "request_context": [
	  "request_id": "",
		"tracing": {
	    "trace_id": "",
	    "span_id": "",
	    "sampled": "False",
		},
	],
}