woocommerce-ios
woocommerce-ios copied to clipboard
[ExPlat] Add A/A test to the app
Closes: #7407
Description
It's a simple change adding A/A experiment name to the app.
Experiment design (internal ref): pbxNRc-1QS-p2
You can test the changes from this Pull Request by:
- Clicking here or scanning the QR code below to access App Center
- Then installing the build number
pr7411-275f1a1
on your iPhone
I just updated the status of this PR to put it on hold for now:
Let's wait for the Abacus setup to be done before adding this test. That way, we can test that the control/treatment assignment is working as expected before merging the PR. There's more about doing that testing here: PCYsg-Fq7-p2#4-testing-the-experiment-code
We need this ExPlat/Abacus setup to be done first:
- Support for making test assignments for the
woocommerceios
platform - Our
woocommerceios
metrics added to Abacus - The test to be fully set up in Abacus (I'll do this step once the first two are done)
I'll come back to review/test this PR once that's done.
Awesome, exciting to see this A/A test close to shipping! And thanks for testing, LGTM. I would recommend making sure users can be assigned treatment
too. Does using the Abacus bookmarklet work to manually assign yourself to the treatment
variation?
For the exposure event, does it fire after the assignment? If not, then we can either remove the exposure event from the experiment model in Abacus or choose a different exposure event. Since this is an A/A experiment, it doesn't matter too much to have or not have an exposure event.
This will count as an experiment code review, so we can check that off after we confirm treatment
also gets assigned.
Thanks for reviewing this!
I would recommend making sure users can be assigned treatment too. Does using the Abacus bookmarklet work to manually assign yourself to the treatment variation?
Yep, I manually assigned myself to the treatment
variation and received the expected assignment:
"variations": {
"woocommerceios_explat_aa_test_202208": "treatment"
},
For the exposure event, does it fire after the assignment?
It does. 👍 For future reference, I checked this by setting breakpoints in ExPlatService.swift:80
(where we handle completion(assignments)
) and in WooAnalytics.swift:183
(where we fire the *_application_opened
event) and confirmed I hit the assignment breakpoint first, and then the Tracks breakpoint.