propulsion icon indicating copy to clipboard operation
propulsion copied to clipboard

feat: add telemetry to the ingestion pipeline

Open nordfjord opened this issue 1 year ago • 0 comments

Adds some basic telemetry to the ingestion pipeline.

All attributes are prefixed with propulsion.

Attribute Description
category The category being processed
stream_name The full stream name being processed
stream_id The id of the stream being processed
batch_size The size of the batch being processed
first_timestamp The receive timestamp of the first event in the batch being handled
lead_time_ms The lead time for processing the batch
Screenshot of trace showing a test run

The above screenshot shows a run of the It doesn't read the tail event again test. It appends 20 events via an equinox decider before starting to poll. You can see it create the checkpoints table, read from it, poll for events, and then fetching two batches before processing a stream batch. Then you see the tailSleepInterval kick in with a wait for 1s before storing the checkpoint and ending the test.

This gives users the opportunity to ship these spans to various observability providers and ask questions like "What's my average lead time for processing category X"

nordfjord avatar Jun 12 '23 01:06 nordfjord