Online performance metrics

[alecgunny]

Part of the appeal of using neural networks for real-time detection is their comparatively small compute footprint. Any publication on their efficacy should show this explicitly by showing how throughput trades off against latency for different batch sizes (since samples earlier in the batch will have extra latency compared to samples later in the batch).

In production, required throughput is actually just decided by your inference sampling rate, so worth also showing performance (in whatever way you want to define performance) as a function of inference sampling rate to get a full illustration of the tradeoffs involved.