Proposal: frint-experiments for A/B testing and more...

[fahad19]

(Intended to be done in a separate repositiory)

What

Guidelines and API spec on how we can run experiments in the UI with FrintJS.

Extracted our usage out of the internal PoC we did recently.

/cc @AlexDudar, @juliamaksimchik, @discosultan, @viacheslaff

Proposed packages

frint-experiments

Will export:

• ExperimentsService
  • will be set as a provider in Apps
  • will provide access to the collection
• ExperimentsCollection: will contain the experiments data
• Experiment: Individual experiment item as a model
• createExperiments: Generates a new configured ExperimentsService class

Example usage:

import { createApp } from 'frint';
import { createExperiments } from 'frint-experiments';

const App = createApp({
  name: 'MyApp',
  providers: [
    {
      name: 'experiments',
      useFactory() {
        const ExperimentsService = createExperiments({
          activate(model, attributes = {}) {
            // implement activation logic
          },

          track(model, eventTags = {}) {
            // implement tracking logic
          },
        });

        return new ExperimentsService([]); // feed it with Experiments collection
      }
    }
  ]
});

The model can have this schema:

// Experiment.js
import { createModel, Types } from 'frint-data';

export default createModel({
  schema: {
    name: Types.string,
    variant: Types.string,
  },
});

The service:

class ExperimentsService {
  constructor(experimentsArray) {
    this.experiments = new ExperimentsCollection(experimentsArray);
  }

  findByName(name) {
    return Experiment;
  }

  findByName$(name) {
    return Observable;
  }

  activate(model, attributes = {}) {
  
  }
  
  track(model, eventTags = {}) {

  }
}

frint-experiments-react

This package will be exporting React components, for implementing our experiments at UI-level.

import React from 'react';
import { Experiment, Variant } from 'frint-experiments-react';

export default function MyComponent(props) {
  return (
    <div>
      <Experiment name="my-experiment-name">
        <Variant when="b" render={() => 'foo'} />
        <Variant when="c" render={() => 'bar'} />
        <Variant render={() => 'default'} />
      </Experiment>
    </div>
  );
}

Experiment component will receive either a name (String) or model (Experiment) as prop.

Benefits

• Declarative experiments
• Always defined at code-level
• No external manipulation with JS code
• Can go for aggressive code-splitting at Component-level, so that users only download the code they need for their own experiments only

[viacheslaff]

It would be good to have automatic activation of the experiment once its variant is accessed. Activation is notification of A/B testing service that the user was exposed to the given experiment. Can we add getter to the Experiment model? If we do this, it'll be transparent for developers whether they are accessing variant in imperative code or declaratively via <Experiment /> component.

Besides activations can be throttled/sent in batches to avoid many XHR requests, but it's an implementation detail.

[markvincze]

What can the when property be set to? Any boolean expression? Or this is outside of the scope of this proposal?

Do you have also already an idea about what data to collect about these experiments and how to collect them?

It might be convenient to support this syntax too (I don't know if it's difficult to do):

  return (
    <div>
      <Experiment model={experiment}>
        <Variant when="a">
          <span>Variant A</span>
        </Variant>
        <Variant when="b" />
          <div>Variant B</div>
        </Variant>
        <Variant />
          <p>Default variant</p>
        </Variant>
      </Experiment>
    </div>
  );

[fahad19]

What can the when property be set to? Any boolean expression? Or this is outside of the scope of this proposal?

The value of when prop will be checked against Experiment model's variant key, which is a Types.string.

It might be convenient to support this syntax too

Sure, we can support it! But if props.children are used with Variant, it means code-splitting will be more difficult then.

Further reading:

• https://reactjs.org/docs/composition-vs-inheritance.html
• https://github.com/tc39/proposal-dynamic-import
• https://webpack.js.org/guides/code-splitting/#dynamic-imports
• https://github.com/thejameskyle/react-loadable

[discosultan]

It would be good to have automatic activation of the experiment once its variant is accessed. Activation is notification of A/B testing service that the user was exposed to the given experiment.

Note that while it would be nice to default to an automatic activation, we will also need a more granular option to cater for cases where the activation must not happen before the subject enters viewport, for example.

[viacheslaff]

One small addition. When talking of A/B testing, as far as I know, variant A represents what we already have. So not using <Variant when="a" render={() => 'foo'} /> will make more sense:

<Experiment model={experiment}>
  <Variant when="b" render={() => 'bar'} />
  <Variant render={() => 'default'} />
</Experiment>

Then from b on, more variants can be added if needed (c, d, etc.)

[mAiNiNfEcTiOn]

Hmmm @asci, it would be a perfect time for you to integrate your concept/idea of DNA/Genomes with your Evolutio, no?

[asci]

Hello,

Thanks for mentioning me, Ricardo! Unfortunately, I think there is no way to integrate Evolutio or its idea with Frint. But I can help with codemod script for auto-removing tests if there will be a request from the core team.

My 5 cents on the topic:

• Is it possible to use the model without React components? For example, you need to make a request or to do some calculations for some case, not just render?
• Any thought about styles and ab\testing? Usually, the test requires some style changes as well. Would be nice to have everything for AB testing under one project with some guides.
• Not clear about activation — when we get data for tests (model content) we can perform activation at that moment (or even do at server time). OR there is no way to split users by experiments\pages at Google Analytics? Also, how it would work during SSR?
• AFAIU it is against Frint to use context types and top-level providers (thinking about Apollo)? If I'm wrong, you can omit the whole part about getting experiment prop.

Cheers!

[fahad19]

Thanks for your questions, @asci! I am responding to each one below:

---

Is it possible to use the model without React components? For example, you need to make a request or to do some calculations for some case, not just render?

Yes. In FrintJS, rendering is an optional thing to do. You first define Apps, that can later be optionally rendered on demand.

The instance of ExperimentsService class will be set as a provider in a FrintJS App, which means you can access it, along with the Collection/Models by doing:

app.get('experiments'); // the ExperimentsService instance

---

Any thought about styles and ab\testing? Usually, the test requires some style changes as well. Would be nice to have everything for AB testing under one project with some guides.

Could you please give me an example scenario? Would help me understand and answer better.

If using CSS Modules, why not just render different Components that can have their own variations in styling?

---

Not clear about activation — when we get data for tests (model content) we can perform activation at that moment (or even do at server time). OR there is no way to split users by experiments\pages at Google Analytics? Also, how it would work during SSR?

Activation/Tracking would require more thoughts. In this proposal, I am limiting the activation/tracking part by defining them in the ExperimentsService interface only.

Based on a particular application's requirements in a given environment (client OR server), we can always decide what to do with the incoming experiment Model.

AFAIU it is against Frint to use context types and top-level providers (thinking about Apollo)? If I'm wrong, you can omit the whole part about getting experiment prop.

FrintJS Apps have Providers, and when the Apps are rendered with frint-react, the App instance is made available in the React's context.

The higher-order component then deals with React's context API and gives us access to the app instance:

import { observe } from 'frint-react';

function MyComponent(props) { 

}

export default observe(function (app) {
  // make props compatible object here
})(MyComponent);

If we really want to enable developers to write less code, we can additionally support a name prop in the <Experiment> Component:

import { Experiment, Variant } from 'frint-experiments-react';

export default function MyComponent(props) {
  return (
    <div>
      <Experiment name="my-experiment-name">
        <Variant when="b" render={() => 'b'} />
        <Variant when="c" render={() => 'c'} />
        <Variant render={() => 'default'} />
      </Experiment>
    </div>
  );
}

The Experiment component's behaviour would then be:

• If model prop is available, use that Experiment model
• If name prop is available, get the model from ExperimentService, and then use that