feat(experiment): Significance for secondary metrics

[neilkakkar]

Problem

Add secondary metrics significance calculation, and add their graphs to result screen as well

See demo: https://posthog.slack.com/archives/C034XD440RK/p1707762026996979

Changes

👉 Stay up-to-date with PostHog coding conventions for a smoother review.

How did you test this code?

unit tests, run locally, cache results from master, then switch branch and see it all run ok

[github-actions[bot]]

Size Change: 0 B

Total Size: 864 kB

ℹ️ View Unchanged

Filename	Size
frontend/dist/toolbar.js	864 kB

_{compressed-size-action}

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[posthog-bot]

📸 UI snapshots have been updated

2 snapshot changes in total. 0 added, 2 modified, 0 deleted:

• chromium: 0 added, 2 modified, 0 deleted (diff for shard 1)
• webkit: 0 added, 0 modified, 0 deleted

Triggered by this commit.

👉 Review this PR's diff of snapshots.

[jurajmajerik]

Clicking on the line graph triggers an error, can you check?

[neilkakkar]

Weird, how can I reproduce? My graphs seem fine 🤔 , can you tell me setup instructions?

[jurajmajerik]

I tested this with an existing experiment, does this help with reproducing?

{ "id": 14, "name": "Exp 14", "description": "Funnel: \"test\" winning", "start_date": "2024-02-07T06:48:44.616000Z", "end_date": null, "feature_flag_key": "exp-14", "feature_flag": { "id": 92, "team_id": 1, "name": "Feature Flag for Experiment Exp 14", "key": "exp-14", "filters": { "groups": [ { "properties": [], "rollout_percentage": 100 } ], "multivariate": { "variants": [ { "key": "control", "rollout_percentage": 50 }, { "key": "test", "rollout_percentage": 50 } ] }, "aggregation_group_type_index": null }, "deleted": false, "active": true, "ensure_experience_continuity": false }, "parameters": { "feature_flag_variants": [ { "key": "control", "rollout_percentage": 50 }, { "key": "test", "rollout_percentage": 50 } ], "recommended_sample_size": 0, "recommended_running_time": null }, "secondary_metrics": [ { "name": "Sec 1", "filters": { "events": [ { "id": "$pageview", "math": "total", "name": "$pageview", "type": "events", "order": 0 } ], "date_to": "2024-02-13T23:59", "display": "ActionsLineGraph", "insight": "TRENDS", "interval": "day", "date_from": "2024-01-30", "entity_type": "events" } } ], "filters": { "events": [ { "id": "exp 14 step 1", "name": "exp 14 step 1", "type": "events", "order": 0 }, { "id": "exp 14 step 2", "name": "exp 14 step 2", "type": "events", "order": 1 } ], "layout": "horizontal", "date_to": "2024-02-07T23:59", "insight": "FUNNELS", "interval": "day", "date_from": "2024-01-24T07:46", "entity_type": "events", "funnel_viz_type": "steps", "filter_test_accounts": false }, "archived": false, "created_by": { "id": 1, "uuid": "018d5f9b-e509-0000-44e1-dac920d35d34", "distinct_id": "IReL69komZk8gor2UNFh7re3aORQFgbQqR6oHpys42X", "first_name": "Juraj", "last_name": "", "email": "[email protected]", "is_email_verified": false }, "created_at": "2024-02-07T06:47:13.632991Z", "updated_at": "2024-02-13T09:30:16.016985Z" }

[jurajmajerik]

And the experiment results:

{ "result": { "insight": [ [ { "action_id": "exp 14 step 1", "name": "exp 14 step 1", "custom_name": null, "order": 0, "people": [], "count": 161, "type": "events", "average_conversion_time": null, "median_conversion_time": null, "breakdown": [ "test" ], "breakdown_value": [ "test" ], "converted_people_url": "/api/person/funnel/?breakdown=%5B%22%24feature%2Fexp-14%22%5D&breakdown_attribution_type=first_touch&breakdown_normalize_url=False&breakdown_type=event&date_from=2024-02-07T06%3A48%3A44.616000%2B00%3A00&explicit_date=true&display=FunnelViz&events=%5B%7B%22id%22%3A+%22exp+14+step+1%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+0%2C+%22name%22%3A+%22exp+14+step+1%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%2C+%7B%22id%22%3A+%22exp+14+step+2%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+1%2C+%22name%22%3A+%22exp+14+step+2%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%5D&entity_type=events&funnel_step_breakdown=%5B%22test%22%5D&funnel_step=1&funnel_viz_type=steps&funnel_window_interval=14&funnel_window_interval_unit=day&insight=FUNNELS&interval=day&layout=horizontal&limit=100&sampling_factor=&smoothing_intervals=1&cache_invalidation_key=llzqS3ep", "dropped_people_url": null }, { "action_id": "exp 14 step 2", "name": "exp 14 step 2", "custom_name": null, "order": 1, "people": [], "count": 98, "type": "events", "average_conversion_time": 0.01020408163265306, "median_conversion_time": 0.0, "breakdown": [ "test" ], "breakdown_value": [ "test" ], "converted_people_url": "/api/person/funnel/?breakdown=%5B%22%24feature%2Fexp-14%22%5D&breakdown_attribution_type=first_touch&breakdown_normalize_url=False&breakdown_type=event&date_from=2024-02-07T06%3A48%3A44.616000%2B00%3A00&explicit_date=true&display=FunnelViz&events=%5B%7B%22id%22%3A+%22exp+14+step+1%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+0%2C+%22name%22%3A+%22exp+14+step+1%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%2C+%7B%22id%22%3A+%22exp+14+step+2%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+1%2C+%22name%22%3A+%22exp+14+step+2%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%5D&entity_type=events&funnel_step_breakdown=%5B%22test%22%5D&funnel_step=2&funnel_viz_type=steps&funnel_window_interval=14&funnel_window_interval_unit=day&insight=FUNNELS&interval=day&layout=horizontal&limit=100&sampling_factor=&smoothing_intervals=1&cache_invalidation_key=llzqS3ep", "dropped_people_url": "/api/person/funnel/?breakdown=%5B%22%24feature%2Fexp-14%22%5D&breakdown_attribution_type=first_touch&breakdown_normalize_url=False&breakdown_type=event&date_from=2024-02-07T06%3A48%3A44.616000%2B00%3A00&explicit_date=true&display=FunnelViz&events=%5B%7B%22id%22%3A+%22exp+14+step+1%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+0%2C+%22name%22%3A+%22exp+14+step+1%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%2C+%7B%22id%22%3A+%22exp+14+step+2%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+1%2C+%22name%22%3A+%22exp+14+step+2%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%5D&entity_type=events&funnel_step_breakdown=%5B%22test%22%5D&funnel_step=-2&funnel_viz_type=steps&funnel_window_interval=14&funnel_window_interval_unit=day&insight=FUNNELS&interval=day&layout=horizontal&limit=100&sampling_factor=&smoothing_intervals=1&cache_invalidation_key=llzqS3ep" } ], [ { "action_id": "exp 14 step 1", "name": "exp 14 step 1", "custom_name": null, "order": 0, "people": [], "count": 131, "type": "events", "average_conversion_time": null, "median_conversion_time": null, "breakdown": [ "control" ], "breakdown_value": [ "control" ], "converted_people_url": "/api/person/funnel/?breakdown=%5B%22%24feature%2Fexp-14%22%5D&breakdown_attribution_type=first_touch&breakdown_normalize_url=False&breakdown_type=event&date_from=2024-02-07T06%3A48%3A44.616000%2B00%3A00&explicit_date=true&display=FunnelViz&events=%5B%7B%22id%22%3A+%22exp+14+step+1%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+0%2C+%22name%22%3A+%22exp+14+step+1%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%2C+%7B%22id%22%3A+%22exp+14+step+2%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+1%2C+%22name%22%3A+%22exp+14+step+2%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%5D&entity_type=events&funnel_step_breakdown=%5B%22control%22%5D&funnel_step=1&funnel_viz_type=steps&funnel_window_interval=14&funnel_window_interval_unit=day&insight=FUNNELS&interval=day&layout=horizontal&limit=100&sampling_factor=&smoothing_intervals=1&cache_invalidation_key=MLyl64M2", "dropped_people_url": null }, { "action_id": "exp 14 step 2", "name": "exp 14 step 2", "custom_name": null, "order": 1, "people": [], "count": 60, "type": "events", "average_conversion_time": 0.0, "median_conversion_time": 0.0, "breakdown": [ "control" ], "breakdown_value": [ "control" ], "converted_people_url": "/api/person/funnel/?breakdown=%5B%22%24feature%2Fexp-14%22%5D&breakdown_attribution_type=first_touch&breakdown_normalize_url=False&breakdown_type=event&date_from=2024-02-07T06%3A48%3A44.616000%2B00%3A00&explicit_date=true&display=FunnelViz&events=%5B%7B%22id%22%3A+%22exp+14+step+1%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+0%2C+%22name%22%3A+%22exp+14+step+1%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%2C+%7B%22id%22%3A+%22exp+14+step+2%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+1%2C+%22name%22%3A+%22exp+14+step+2%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%5D&entity_type=events&funnel_step_breakdown=%5B%22control%22%5D&funnel_step=2&funnel_viz_type=steps&funnel_window_interval=14&funnel_window_interval_unit=day&insight=FUNNELS&interval=day&layout=horizontal&limit=100&sampling_factor=&smoothing_intervals=1&cache_invalidation_key=MLyl64M2", "dropped_people_url": "/api/person/funnel/?breakdown=%5B%22%24feature%2Fexp-14%22%5D&breakdown_attribution_type=first_touch&breakdown_normalize_url=False&breakdown_type=event&date_from=2024-02-07T06%3A48%3A44.616000%2B00%3A00&explicit_date=true&display=FunnelViz&events=%5B%7B%22id%22%3A+%22exp+14+step+1%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+0%2C+%22name%22%3A+%22exp+14+step+1%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%2C+%7B%22id%22%3A+%22exp+14+step+2%22%2C+%22type%22%3A+%22events%22%2C+%22order%22%3A+1%2C+%22name%22%3A+%22exp+14+step+2%22%2C+%22custom_name%22%3A+null%2C+%22math%22%3A+null%2C+%22math_property%22%3A+null%2C+%22math_hogql%22%3A+null%2C+%22math_group_type_index%22%3A+null%2C+%22properties%22%3A+%7B%7D%7D%5D&entity_type=events&funnel_step_breakdown=%5B%22control%22%5D&funnel_step=-2&funnel_viz_type=steps&funnel_window_interval=14&funnel_window_interval_unit=day&insight=FUNNELS&interval=day&layout=horizontal&limit=100&sampling_factor=&smoothing_intervals=1&cache_invalidation_key=MLyl64M2" } ] ], "filters": { "breakdown": [ "$feature/exp-14" ], "breakdown_attribution_type": "first_touch", "breakdown_normalize_url": false, "breakdown_type": "event", "date_from": "2024-02-07T06:48:44.616000+00:00", "explicit_date": "true", "display": "FunnelViz", "events": [ { "id": "exp 14 step 1", "type": "events", "order": 0, "name": "exp 14 step 1", "custom_name": null, "math": null, "math_property": null, "math_hogql": null, "math_group_type_index": null, "properties": {} }, { "id": "exp 14 step 2", "type": "events", "order": 1, "name": "exp 14 step 2", "custom_name": null, "math": null, "math_property": null, "math_hogql": null, "math_group_type_index": null, "properties": {} } ], "entity_type": "events", "funnel_viz_type": "steps", "funnel_window_interval": 14, "funnel_window_interval_unit": "day", "insight": "FUNNELS", "interval": "day", "layout": "horizontal", "limit": 100, "sampling_factor": "", "smoothing_intervals": 1 }, "probability": { "control": 0.005229999999999957, "test": 0.99477 }, "significant": true, "significance_code": "significant", "expected_loss": 9.70199677235683e-05, "variants": [ { "key": "control", "success_count": 60, "failure_count": 71 }, { "key": "test", "success_count": 98, "failure_count": 63 } ] }, "last_refresh": "2024-02-13T13:22:19.606735Z", "is_cached": true }

[posthog-bot]

This PR hasn't seen activity in a week! Should it be merged, closed, or further worked on? If you want to keep it open, post a comment or remove the stale label – otherwise this will be closed in another week.

[neilkakkar]

Riight the problem is with the hogql-insight-trends flag, can you review once more please with that flag disabled? Will fix the flag issues in a follow up