cleaver
cleaver copied to clipboard
Track individual participation INCOMPLETE
I'm making this PR just to have a place for discussion -- its incomplete because it does the RedisBackend
and SQLAlchemyBackend
behave differently at the moment. Also, depending on the vision/roadmap for cleaver, this may not be desired and can certainly live in a fork.
This PR adds the user identity as a parameter to cleaver API calls for mark_participant
and mark_conversion
Passing the identity allows the backend to keep track of individual participation so that experiments may be conducted in situations where conversion events might happen multiple times. Basically, the backend can keep track of the number of unique participants and score one and only one conversion for each unique participant.
Otherwise, if the conversion event can happen more than once, its impossible to distinguish the true number of conversions for each variant (e.g., one individual scoring many conversions vs several individuals scoring several conversions) and therefore difficult to draw conclusions from the experiment results.
This different behavior could be enabled/disabled through an option passed to the backend so that cleaver defaults to the original behavior. It was pretty trivial to implement in the RedisBackend
, but would require new models and some additional complications in the SQLAlchemyBackend