humanmade
Make A/B test queries only track unique users & conversions
Right now the aggregated stats collected for A/B tests every hour are for any unique page session rather than unique visitor. This isn't really the industry standard and is a limitation of sampling and aggregating data over time.
We could sample by unique visitors only but it would be unique visitors for an hour at a time, there might be cross-over between hourly sampling.
Needs some research to understand / how if this can be done. Querying up to 90 days worth of data, although an option, could be very slow indeed.
We could possibly exclude users once they've been exposed to a test once, but that would exclude them if they were to convert at a later date.
We could increase the sampling window, so still take it every hour, but also a daily aggregate or running aggregate from all the available data.