posthog icon indicating copy to clipboard operation
posthog copied to clipboard

feat(batch-exports): Add HTTP Batch Export destination

Open bretthoerner opened this issue 1 year ago • 3 comments

Problem

We do region migrations via a local script when we already have a nice Batch Export feature.

Changes

This adds an HTTP destination to Batch Exports. The plan is to keep it hidden from users (for now?), although any UI work will be done in a separate PR.

Just saying "HTTP Destination" is vague, as there are infinite possible HTTP APIs. We currently hardcode the payloads to be in the shape required for PostHog's /batch capture endpoint. In the future we could support different formats, although figuring out how to template arbitrary HTTP payloads (and content types?) will get weird.

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

Added tests.

bretthoerner avatar Feb 13 '24 18:02 bretthoerner

TestDashboard.test_dashboard_duplication failure is unrelated: https://posthog.slack.com/archives/C0113360FFV/p1707820207148159

bretthoerner avatar Feb 13 '24 20:02 bretthoerner

Similar to S3, we'll want to heartbeat, track progress, and resume from progress for this destination.

tomasfarias avatar Feb 14 '24 13:02 tomasfarias

OK, I should have covered the important bits (such as heartbeating). I'd like to merge this soon to do some testing and continue with other pieces in incremental PRs.

bretthoerner avatar Feb 14 '24 18:02 bretthoerner

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

  • ‼️ TypeError: HttpBatchExportInputs.init() missing 1 required positional argument: 'batch_export_id' posthog.temporal.batch_exports.http_batch_expor... View Issue
  • ‼️ InvalidURL: us.posthog.com aiohttp.client_reqrep in update_host View Issue
  • ‼️ TypeError: Only io.IOBase, multidict and (name, file) pairs allowed, use .add_field() for passing more compl... aiohttp.formdata in add_fields View Issue

Did you find this useful? React with a 👍 or 👎

sentry[bot] avatar Feb 19 '24 14:02 sentry[bot]