posthog icon indicating copy to clipboard operation
posthog copied to clipboard

feat(experiments): new Bayesian stats module

Open andehen opened this issue 7 months ago • 3 comments

Changes

  • A fresh new implementation of the Bayesian method for A/B testing
  • Follows the same file and object structure as the newly added frequentist method
  • Most notably, the new implementation now models the delta (difference between treatment and control) directly, rather than control and treatment separately
  • Sample variance is also used to better adapt to the real data which results in more reasonable credible intervals

I'm probably going to make more changes to this module in the near future, but I think this is a good enough starting point to be integrated into the query runner.

The idea for integration is to put it behind a feature flag and bump the stats_version number such that it only applies to new experiments. We can then test it out internally and iterate on it before a full release is done.

How did you test this code?

  • unit tests added

andehen avatar Jun 12 '25 20:06 andehen

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

  • chromium: 0 added, 2 modified, 0 deleted (wasn't pushed!)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot avatar Jun 16 '25 12:06 posthog-bot

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot avatar Jun 16 '25 12:06 posthog-bot

📸 UI snapshots have been updated

1 snapshot changes in total. 0 added, 1 modified, 0 deleted:

[!CAUTION]

Detected flapping snapshots

These snapshots have auto-updated more than once since the last human commit:

  • scenes-app-insights-funnels--funnel-top-to-bottom-edit--dark.png (chromium, shard 2)

The flippy-flappies are deadly and must be fixed ASAP. They're productivity killers. Run pnpm storybook locally and make the fix now. (Often, the cause is ResizeObserver being used instead of the better CSS container queries.)

  • chromium: 0 added, 1 modified, 0 deleted (diff for shard 2)
  • webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

posthog-bot avatar Jun 16 '25 12:06 posthog-bot