North star metrics and success indicators for MVP

[sdimran]

We need to define our north star metrics as well as KPI's for MVP so that we understand how these factors will affect our next steps of the product.

Dependency

• [ ] #94
• [x] #179

Action items

We need answers to the following questions

• [ ] How will we know that users are finding value from CTJ
• [ ] what KPI's do we need to include on the landing page and search page to understand where users are concentrating their actions

[sdimran]

As discussed, our north star MVP will be the turnaround rate for users coming to civictechjobs.org and them finding roles that fit their criteria. how many users came and how many found roles?

Additional KPI's to consider:

1. number of users reaching civictechjobs.org for the first time
2. number of users selecting call to action
3. number of users attending onboarding after selecting call to action

these KPI's should be measured using google analytics (#94)

[stephaniestahlberg]

In addition to the number of users in the categories that @sdimran mentioned above, track the share of users in these categories. So that we know if both the number and share are increasing.

Other ideas for metrics, so we can compare the performance of the current process to the new CTJ website:

• Number and share of new volunteers who attended onboarding but never joined a project
• Time it takes for new volunteers to join a project after attending onboarding (average time, median, standard deviation)

Can we get data on these? Perhaps there are logs or databases where we have people's onboarding date and date when they first appear on a project's roster? I would be happy to analyze these.

[stephaniestahlberg]

Some ideas for metrics for CMS:

• Amount of time it takes to post a new role with the old and new processes (I don't expect we have existing data on this. Would need to collect via usability testing so we can observe and time PMs posting a new role).
• Amount of time it takes to fill a role (could we possibly have a github log of when roles are posted and taken down? Acknowledging that posts may not be removed right away after a role is filled. And some roles are filled without being posted. So the data on this does not sound very accurate).
• Qualitative and/or survey evidence from PMs describing their experience with the old and new processes.

[stephaniestahlberg]

What are our success indicators, which show that CTJ works better than the old process? We cannot use many of the more traditional indicators, such as time spent on task, task success rate, back button clicks, error rate, etc., derived from usability tests and/or website analytics, because these are not tracked for the current process that volunteers go through to join HfLA and projects. But we can collect other similar indicators via surveys and interviews, by asking people directly how the process worked for them.

For example:

• Usability Indicators: frustration level with the process (e.g. 5-point scale questions), number of unique problems/challenges PMs and leads identify, share of users who identify each problem, scores on System Usability Scale (SUS)
• Efficiency Indicators: how long it took to join Hack for LA, how long it took to join a project, how long it takes to post a role, how long it takes to fill a role, number of ineligible candidates reaching out to PMs and leads
• Effectiveness Indicator: share of new onboarded volunteers who actually join a project
• Visibility Indicator: number of page views (it seems we can get this on the Github roles page as well)

[stephaniestahlberg]

As we conduct Phase 1 of the CMS research and speak with PMs and Leads, we will ask them to identify the main pain points with the current Hack for LA process for posting roles and recruiting team members. These responses will help us create metrics to compare the current process with the new CTJ process. Each system's performance according to these metrics will be measured by surveys: for the current process, the survey can go out as soon as we define the metrics and create the survey, and for the new process with the CTJ MVP, a survey will be sent out once the page is live and PMs and Leads have had a chance to post roles and recruit using the new process.