Pipeline Metrics: End-to-End processing latency

[rghetia]

trafficstars

This issue specifically covers end-to-end latency metric which is a part of overall pipeline metrics (#484).

There are few things to consider here.

Should there be one data point for each metric/span or should it be for a batch?

• Batch is preferred because metrics in a batch would have the same latency.

What happens when multiple batches are combined?

• It should keep track of receive time for each batch. It should then record latency for original batch. For example, if batch A and B are combined into C then when C is exported it should record two latency (Tc-Ta) and (Tc-Tb)

What if batches are split?

• If the batches are split it should again record latency for each split batches. For example if batch A is split into two batches B and C then record latency (Tb-Ta) and (Tc-Ta)

What labels should it have?

• Receiver and Exporter at minimum. This should define the pipeline. Processors may not be required.

What if the element is dropped/filtered?

• May be record latency when it is dropped/filtered. If this is the case then instead of Receiver/Exporter label it should be Begin/End label. Begin could be any receiver and End could be exporter or a processor.

What should be the type of metric?

• Histogram (Distribution)

What about latency of individual stage in the pipeline?

• This would help to debug/isolate the problem when the end-to-end latency is high.
• The latency can be recorded using same measure with Start and End label containing the pipeline/stage name.