issue/7097-explat-aa-test

[nbradbury]

trafficstars

Closes: #7097 - this tiny PR simply adds woocommerceandroid_explat_aa_test_202208 to our experiments. It's submitted as a draft until Abacus has been setup.

Experiment design (internal ref): pbxNRc-1QS-p2

Related iOS PR

• [x] I have considered if this change warrants user-facing release notes and have added them to RELEASE-NOTES.txt if necessary.

[wpmobilebot]

You can test the changes on this Pull Request by downloading an installable build, or scanning this QR code:

[rachelmcr]

The A/A experiment is set up in Abacus in staging (it's not yet running).

While the experiment is in staging, we should be able to test it while logged in to an a8c account in the app. When the app opens, it should make a request to the ExPlat endpoint /wpcom/v2/experiments/0.1.0/assignments/woocommerceandroid to retrieve the user's assignment for the experiment.

However, I'm not seeing any request being made to that endpoint. I'm not familiar with the ExPlat implementation on Android; could someone who knows more about it take a look to see if we're missing something?

[nbradbury]

However, I'm not seeing any request being made to that endpoint. I'm not familiar with the ExPlat implementation on Android; could someone who knows more about it take a look to see if we're missing something?

@RenanLukas It looks like you added ExperimentationModule to WCAndroid. Can you help with the above?

[RenanLukas]

Hey @nbradbury

As far as I can see, the request to the /assignments endpoint is made here and has to meet certain conditions. I suppose the request is not showing because getVariation was not called? This class was originally implemented by @renanferrari, so maybe he sees something I'm not seeing 🙂

Also, @ThomazFB tested my PR back then, so maybe he faced something similar? 🤔

[nbradbury]

@ThomazFB @rachelmcr I've added the code we discussed on Slack and I've verified the call to /wpcom/v2/experiments/0.1.0/assignments/woocommerceandroid/ is being made, however I'm not sure what to make of the response:

{
  "variations": {
    "woocommerceandroid_explat_aa_test_202208": null
  },
  "ttl": 60,
  "debug": {
    "backend_aa_result": "request not sampled"
  }
}

Is "request not sampled" expected since this is an A/A test?

[peril-woocommerce[bot]]

	Warnings
:warning:	PR is not assigned to a milestone.

Generated by :no_entry_sign: dangerJS

[ThomazFB]

@nbradbury, I believe the data you're looking for should be returned inside the variations object. What's being returned as null there should be the variation name. That is what we should expect as the value to effectively execute the A/B test inside the app.

[nbradbury]

@nbradbury, I believe the data you're looking for should be returned inside the variations object. What's being returned as null there should be the variation name. That is what we should expect as the value to effectively execute the A/B test inside the app.

Sorry, I'm not sure what you mean. Is there something I need to change?

[ThomazFB]

Sorry, I'm not sure what you mean. Is there something I need to change?

I believe no change is required from your side. I was just clarifying what you should expect from that response. I don't know in what scenarios the ExPlat API will return null for a given variation request 😅. Maybe null is for the default one that returns as control in the end? That's what I got within the app when I tested this experiment:

As you can see, the name of the variation is not null, is control.

[rachelmcr]

Maybe null is for the default one that returns as control in the end?

Yep, that's right! Sorry for not mentioning that earlier. The API will return null for the control group and treatment for the treatment group. (Documentation here: PCYsg-Fq9-p2#why-am-i-receiving-a-null-assignment-group)

You can test both scenarios by manually assigning yourself (using your a8c account) to one or the other groups from within the Abacus experiment setup (linked at the top of the experiment design: pbxNRc-1QS-p2). In the Audience section of the experiment setup, you can go to Variations > Manual Assignment and use the bookmarklet to assign yourself to either the control or treatment group.