apache
end to end tracking lag feature removed unintentionally
https://github.com/apache/pinot/pull/16783 removed this feature thinking it is not used, but it was a functional feature which we use in stripe
https://github.com/apache/pinot/pull/16783#issuecomment-3498367846
Thanks for raising it. @priyen-stripe if the value is only being set in internal plugin shouldn't the metric itself being created in the internal plugin itself? Is it because it's hard to compute metric in plugin given that existing metric code in pinot is not extensible and modular?
I will add it back since it was already present but I am not sure if it's correct way to do it since the metric will not be useful for all OSS users but stripe.
@noob-se7en , firstStreamIngestionTimeMs, by its name, implies its the timestamp of the first ingestion when data travels across multiple topics - its useful for multi hops in streams before making it to pinot.
You cannot do this in the plugin because we only care about the timestamp for data that pinot successfully ingests. our low-level consumer only streams in data and passes it to pinot via the messageBatch interface; pinot may filter it out or do any amount of things with it after wards and we want the lag tracking to be from that point, not from when our client consumes the data from the stream.
it's the same reason ingestion lag is tracked for data that is consumed only (indexed on the segment and when the update function in IngestionDelayTracker is called)
it is also not correct to assume it is not useful for OSS users; there's no way to know if someone else was using this just like we were. It was a supported parameter in the custom stream interface
and ty for planning to add it back
