charon icon indicating copy to clipboard operation
charon copied to clipboard

Move instrumenting of rare events to tracker

Open corverroos opened this issue 2 years ago • 0 comments

Problem to be solved

We currently track the number of duty exit partial signatures in parsigdb by pubkey. This has a few problems:

  • Cannot distinguish between exits submitted for different epoch (will not aggregate correctly).
  • Tracker is a better place for tracking stuff.
  • We do not instrument when exits have been aggregated and broadcasted

Proposed solution

Move instrumenting of "rare" events to tracker:

  • Add a core_tracker_rare_event_total[duty, slot, pubkey, event_source] counter
  • Note that this has risk of high cardinality, but since we will only instrument actual rare events, this should only result in a few time series.
  • Rename tracker component to source.
  • These events are instrumented when event is processed/seen, not on duty expiry.
  • Instrument the following events:
    • duty.Type==DutyExit && source==(parsigex || validatorap i| sigagg)
    • duty.Type==DutyBuilderRegistration && (parsigex || validatorap i| sigagg)/
  • Remove old metric
  • Update simnet dashboard with new metric

Although the metric is very generic, this will allow easy tracking any rare event in future.

Out of Scope

Update cdvn repo dashboard with exit panel

corverroos avatar Aug 17 '22 06:08 corverroos