jitsu icon indicating copy to clipboard operation
jitsu copied to clipboard

Add performance metrics

Open xtreding opened this issue 4 years ago • 0 comments

Problem

At present Jitsu doesn't measure performance, but it is nice to have feature: know about what time does an event need to be written into DWH.

Solution

Add Prometheus Histogram (percentile) with processing time values (separate by push/pull event type). The metric should be calculated: ${event written to DWH time} - ${event start processing time}. For pulled events start processing time will start for example in ResultSaver.Consume(). For pushed events start processing time will start in StreamingWorker after dequeuing and in PeriodicUploader after reading files payload. Let's save global metrics as well as metrics per destination.

xtreding avatar Nov 19 '21 09:11 xtreding