propulsion
propulsion copied to clipboard
feat: add telemetry to the ingestion pipeline
Adds some basic telemetry to the ingestion pipeline.
All attributes are prefixed with propulsion.
Attribute | Description |
---|---|
category |
The category being processed |
stream_name |
The full stream name being processed |
stream_id |
The id of the stream being processed |
batch_size |
The size of the batch being processed |
first_timestamp |
The receive timestamp of the first event in the batch being handled |
lead_time_ms |
The lead time for processing the batch |
The above screenshot shows a run of the It doesn't read the tail event again
test. It appends 20 events via an equinox decider before starting to poll. You can see it create the checkpoints table, read from it, poll for events, and then fetching two batches before processing a stream batch. Then you see the tailSleepInterval kick in with a wait for 1s before storing the checkpoint and ending the test.
This gives users the opportunity to ship these spans to various observability providers and ask questions like "What's my average lead time for processing category X"