posthog
posthog copied to clipboard
chore(session-recordings): separate topics for events as recordings
To decouple session recording processing from events, we produce to a separate topic. At the moment we still use the same consumer group to consuming both topics, but it should at least allow for pulling larger batches of analytics events at a time and have little more fair scheduling between them.
The key changes here are:
- send $snapshot and $performance_event events to a separate topic
session_recording_events
- use a separate consumer with a separate group id for consuming from the
topic
above
TODO: add test to verify resilient to non-retriable Kafka / postgres failures.
Problem
Changes
👉 Stay up-to-date with PostHog coding conventions for a smoother review.
How did you test this code?
As discussed this morning, we'll want to keep analytic events alone in their "high SLA target" topic, so we'll need to also move away the new web performance data type. Shall we co-mingle them with snapshots and then have that topic a "low SLA target" one, or have a third one for web perf?
Shall we co-mingle them with snapshots and then have that topic a "low SLA target" one, or have a third one for web perf?
They are colocated for now, leaving that third topic as a potential further improvement.