Mystery User Reviewer

[stackatron]

Solves the same problem as https://github.com/blockstack/app-mining/issues/174 with a different solution.

Overall problem:

• Solving challenges for user retention is taking longer than anticipated.
• App Miners are concerned that abandoned apps are ranking high because the reviewers do not account for app improvements.

What we need is a way to reward meaningful progress on apps. This is hard to objectively measure. And if we did objectively measure it, it would likely be game-able.

Game-able objective measures:

Here are the suggestions from the proof-of-progress thread simply to illustrate the point:

• Posting tweets: Obviously game-able.
• Fixing bugs: Bugs can be purposely introduced and then fixed.
• Adding new features: 100 new features in each app does not really equate to apps getting better.
• Doing partnerships: Interesting, but it is easy to imagine bogus/pointless partnerships. Furthermore, how would a reviewer verify such a thing as partnerships?
• Our true objective:

In my opinion our true objective is to reward apps that provide value to users as measured by retention. Since we are delayed on retention, I'll propose a temporary, secondary objective: Reward apps that make quality improvements that benefit users. This is a subjective goal, and so I suggest we use subjective measures for scoring.

Mystery User Reviewer

@jcnelson thought of this reviewer. He was inspired by Mystery Shoppers that provide quality metrics for retail experiences.

Bootup

• PBC finds a Mystery User Reviewer
• PBC and Mystery User Reviewer create a master list of Testers. Identities will be hidden, but we will publish a list of demographic attributes to ensure we have a non-biased population.
• Community will review the list and make adjustments until we agree the testers represent a varied and fair population to test with.

Monthly run

Mystery User Reviewer uses random selection to pick 5 Testers. Each Tester will:

• Receive a full list of apps using App.co data: Name, icon, description. The list order is randomized for each reviewer, each month.
• "View" the entire list.
• "Select" apps to test. The Testers will use whatever criteria they subjectively feel like to pick which apps get tested. The instruction could read "Pick apps you feel are valuable."
• For each app tested, if the tester feels: -- The app works as expected and is valuable, they will add to a list of "liked" apps. -- The app is broken, does not perform as advertised, or simply isn't valuable to them, they will add to a list of "disliked" apps.
• Tester continues testing until they have 10 liked apps, and 10 disliked apps.
• Each Tester shares their lists with the Mystery User Reviewer.

Scoring

Tallying likes (+1) and dislikes (-1) across all apps to create raw score.

Abstract considerations:

• This system has some downsides, but I think it would encourage:
• Iteration and polish on app names, icons, descriptions.
• App devs choosing to focus on apps that have high value to large cohorts of users.
• Bug free, functional, and maintained apps.
• Discourage exaggeration and lying.
• Many apps will not be reviewed. This reviewer includes a "race to be noticed enough to be reviewed" component.

On the process side of things:

• Unlike the user retention reviewer, this reviewer is ready to go with no new tech.
• We can shut down this reviewer once retention metrics are working properly.

[joshthegreatavenue]

Mysterious reviewer sounds like a good idea in theory but here is the catch.

1. Are the mysterious reviewer alike TMUI?
2. How can we ensure they have no conflict of interest among the Dapps they are reviewing ?
3. Are these mysterious reviewer users themselves?
4. There must be at least 50 or more Mysterious person in this reviewer since I am guessing they are probably normal people. More people equates to better score distribution.

[joberding]

I agree with Josh's point on Mystery Shoppers being users. That should be verified. We have had a problem with TMUI providing testers who claim that they are a specific category of users but are not.

[stackatron]

no conflict of interest

This is explained in the proposal. I know it is not perfect, but there is some attempt to review the people.