posthog icon indicating copy to clipboard operation
posthog copied to clipboard

chore(session-recordings): separate topics for events as recordings

Open hazzadous opened this issue 2 years ago • 2 comments

To decouple session recording processing from events, we produce to a separate topic. At the moment we still use the same consumer group to consuming both topics, but it should at least allow for pulling larger batches of analytics events at a time and have little more fair scheduling between them.

The key changes here are:

  1. send $snapshot and $performance_event events to a separate topic session_recording_events
  2. use a separate consumer with a separate group id for consuming from the topic above

TODO: add test to verify resilient to non-retriable Kafka / postgres failures.

Problem

Changes

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

hazzadous avatar Jan 11 '23 18:01 hazzadous

As discussed this morning, we'll want to keep analytic events alone in their "high SLA target" topic, so we'll need to also move away the new web performance data type. Shall we co-mingle them with snapshots and then have that topic a "low SLA target" one, or have a third one for web perf?

xvello avatar Jan 12 '23 12:01 xvello

Shall we co-mingle them with snapshots and then have that topic a "low SLA target" one, or have a third one for web perf?

They are colocated for now, leaving that third topic as a potential further improvement.

hazzadous avatar Jan 13 '23 16:01 hazzadous