Postgres unavailable events go to DLQ without any retries

[tiina303]

Bug description

For various reasons we can be in a situation for a short time period we can't write to postgres, e.g. maintenance, connection error, ... Currently what happens is we send the event to Dead Letter Queue. We should add some retries, so short Postgres unavailability won't cause us to lose events.

How to reproduce

1. make postgres read only for a brief time
2. see that events trying to change/update persons data went to DLQ

Environment

• [x] PostHog Cloud
• [x] self-hosted PostHog (ClickHouse-based), version/commit: please provide
• [x] self-hosted PostHog (Postgres-based, legacy), version/commit: please provide

Additional context

Thank you for your bug report – we love squashing them!

[smartpierre]

Hi, we made a quick upgrade on our PostgreSQL server and we're experiencing this issue (DLQ doesn't flush). While I understand there's nothing in place to automatically retry the events, is there a way to do this manually ? Thanks !

[tiina303]

Hi, if you would like to do this manually on your self-hosted instance you certainly can. Something to be aware is that for person properties overriding we don't look at timestamps, so if you send some of these events now, then they might get the property values that were old (you can just look through the data to see if that's a concern on your side or not).

So how to do it:

1. get the data from DLQ table - you can just query the table in clickhouse
2. send those events to PostHog (I suggest writing a small script for this, e.g. in python & using the timestamps from the original events)

Let me know if you have any questions :)

[smartpierre]

Alright, I'll try ! Thanks :)

[tiina303]

we have retries now