posthog
posthog copied to clipboard
feat(batch-exports): Add HTTP Batch Export destination
Problem
We do region migrations via a local script when we already have a nice Batch Export feature.
Changes
This adds an HTTP destination to Batch Exports. The plan is to keep it hidden from users (for now?), although any UI work will be done in a separate PR.
Just saying "HTTP Destination" is vague, as there are infinite possible HTTP APIs. We currently hardcode the payloads to be in the shape required for PostHog's /batch capture endpoint. In the future we could support different formats, although figuring out how to template arbitrary HTTP payloads (and content types?) will get weird.
👉 Stay up-to-date with PostHog coding conventions for a smoother review.
How did you test this code?
Added tests.
TestDashboard.test_dashboard_duplication failure is unrelated: https://posthog.slack.com/archives/C0113360FFV/p1707820207148159
Similar to S3, we'll want to heartbeat, track progress, and resume from progress for this destination.
OK, I should have covered the important bits (such as heartbeating). I'd like to merge this soon to do some testing and continue with other pieces in incremental PRs.
Suspect Issues
This pull request was deployed and Sentry observed the following issues:
- ‼️ TypeError: HttpBatchExportInputs.init() missing 1 required positional argument: 'batch_export_id'
posthog.temporal.batch_exports.http_batch_expor...View Issue - ‼️ InvalidURL: us.posthog.com
aiohttp.client_reqrep in update_hostView Issue - ‼️ TypeError: Only io.IOBase, multidict and (name, file) pairs allowed, use .add_field() for passing more compl...
aiohttp.formdata in add_fieldsView Issue
Did you find this useful? React with a 👍 or 👎