census-instrumentation
The exporter API should allow for observation at the recording layer.
The division of the recording API from the exporter API prevents many types of data analysis. Ideally, the exporter should be allowed to "subscribe" to events pushed at the record API (e.g. observe the recording of individual latencies).
The current intermediary view does not adequately cover sophisticated analysis requirements. Moreso, as analysis (both online and for storage) evolve, any intermediary form is likely to prove an obstacle to innovation.
An exporter should be able to attach to "record API" invocations to maintain more robust representations of arriving measurements to power more complete delivery to TSDBs and streaming systems.
@postwait thanks for your feedback. It would be nice to know what kind of sophisticated analysis are you referring to.
Do you think you need to inject yourself on the critical path (sync)? Or just async is fine (see all raw measurements async)?
The types of statistics we'd like to end up calculating are modaliity extractions, quantiles, IQRs, inverse quantiles, distribution shape analysis, but that's just today -- it's an open research area as well.
For most of our specific needs seeing all (100% sampling) raw measurements async (up to say 1ms late) would be sufficient, but our current inline analysis is a flat 9ns and as such it often ends up being more overhead to "get them async" than it is to process them inline. There's only one type of analysis that we do that requires sync support (or nanosecond timing of events) is arrival rate analysis. This last type is hard because even with the metrics framework is sync, the consumer on the metric publication side often has delaying, batching, or ganging that makes that analysis very hard to do right.
I'm happy to explore this in more detail if you have questions.
Hi @postwait is any of this code open source. The 9ns makes me very curious 👍
Some of it. I think the go variant is a bit slower. But, a lot of our stuff is powered by storing the data in an HDR histogram. You can find libcircllhist as a reference.
On Thu, Dec 20, 2018, 3:53 PM Bogdan Drutu <[email protected] wrote:
Hi @postwait https://github.com/postwait is any of this code open source. The 9ns makes me very curious 👍
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/census-instrumentation/opencensus-specs/issues/216#issuecomment-449180846, or mute the thread https://github.com/notifications/unsubscribe-auth/AAUfhPizihWvVrnrRTbqdGoRwMhsljxpks5u7CLzgaJpZM4Yt1-k .