woocommerce-ios icon indicating copy to clipboard operation
woocommerce-ios copied to clipboard

[ExPlat] Add A/A test to the app

Open ealeksandrov opened this issue 2 years ago • 4 comments

Closes: #7407

Description

It's a simple change adding A/A experiment name to the app.

Experiment design (internal ref): pbxNRc-1QS-p2

ealeksandrov avatar Aug 03 '22 11:08 ealeksandrov

You can test the changes from this Pull Request by:

  • Clicking here or scanning the QR code below to access App Center
  • Then installing the build number pr7411-275f1a1 on your iPhone
If you need access to App Center, please ask a maintainer to add you.

wpmobilebot avatar Aug 03 '22 11:08 wpmobilebot

I just updated the status of this PR to put it on hold for now:

Let's wait for the Abacus setup to be done before adding this test. That way, we can test that the control/treatment assignment is working as expected before merging the PR. There's more about doing that testing here: PCYsg-Fq7-p2#4-testing-the-experiment-code

We need this ExPlat/Abacus setup to be done first:

  1. Support for making test assignments for the woocommerceios platform
  2. Our woocommerceios metrics added to Abacus
  3. The test to be fully set up in Abacus (I'll do this step once the first two are done)

I'll come back to review/test this PR once that's done.

rachelmcr avatar Aug 03 '22 12:08 rachelmcr

Awesome, exciting to see this A/A test close to shipping! And thanks for testing, LGTM. I would recommend making sure users can be assigned treatment too. Does using the Abacus bookmarklet work to manually assign yourself to the treatment variation?

For the exposure event, does it fire after the assignment? If not, then we can either remove the exposure event from the experiment model in Abacus or choose a different exposure event. Since this is an A/A experiment, it doesn't matter too much to have or not have an exposure event.

This will count as an experiment code review, so we can check that off after we confirm treatment also gets assigned.

aaronyan avatar Aug 10 '22 17:08 aaronyan

Thanks for reviewing this!

I would recommend making sure users can be assigned treatment too. Does using the Abacus bookmarklet work to manually assign yourself to the treatment variation?

Yep, I manually assigned myself to the treatment variation and received the expected assignment:

"variations": {
	"woocommerceios_explat_aa_test_202208": "treatment"
},

For the exposure event, does it fire after the assignment?

It does. 👍 For future reference, I checked this by setting breakpoints in ExPlatService.swift:80 (where we handle completion(assignments)) and in WooAnalytics.swift:183 (where we fire the *_application_opened event) and confirmed I hit the assignment breakpoint first, and then the Tracks breakpoint.

rachelmcr avatar Aug 10 '22 17:08 rachelmcr