posthog
posthog copied to clipboard
PostHog
feat: set export batch size based on plugin settings
Problem
Sister PR to https://github.com/PostHog/plugin-scaffold/pull/52 and https://github.com/PostHog/s3-export-plugin/pull/22: exports to blob storage (s3 / gcs / snowflake) should use a higher batch size than the default 500, to improve throughput and reduce cost.
Changes
https://github.com/PostHog/posthog/pull/13481 hard an ad-hoc solution for s3, we're generalizing it, based on a signal given by the plugin itself.
Cannot deploy yet until the s3 changes are released to prod-us, in the meantime I'll leave it in draft.
How did you test this code?
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.
This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.
Unblocked by merging https://github.com/PostHog/posthog/pull/13855 and manually fixing the S3 plugin URL on prod-eu. Rebased and ready for final review.