jitsu
jitsu copied to clipboard
Add performance metrics
Problem
At present Jitsu doesn't measure performance, but it is nice to have feature: know about what time does an event need to be written into DWH.
Solution
Add Prometheus Histogram (percentile) with processing time values (separate by push/pull event type). The metric should be calculated: ${event written to DWH time} - ${event start processing time}. For pulled events start processing time will start for example in ResultSaver.Consume(). For pushed events start processing time will start in StreamingWorker after dequeuing and in PeriodicUploader after reading files payload. Let's save global metrics as well as metrics per destination.