Usage: should 'meetings' be monthly, rather than cumulative?

[avivapinchas]

Issue - Enhancement

Opening an issue to collect feedback on whether having "meetings" be cumulative is misleading. This issue is just meant to collect signal and feedback from users.

Acceptance Criteria (optional)

Parabol team is clear on whether users expect meetings to be monthly in the chart. A design or implementation issue has been created, if we intend to make a change.

Estimated effort: X points (see CONTRIBUTING.md)

[nickoferrall]

The current aim of the Usage Stats, in my opinion, is to show free-tier companies that they have been using and getting utility from Parabol. If we show monthly stats, the chart might go down and to the right if it's the holiday season or they previously created a bunch of test meetings.

However, for companies that are already paying to use Parabol, I do think it would be useful. Larger orgs, more suited for the Pro & Enterprise tiers, are more likely to care about detailed Usage Stats as it's harder to get a feel for a 1,000 person org's meeting cadence vs 10 person, and the data is more significant at that level.

I propose:

• We continue to show cumulative usage stats to free-tier users
• Monthly usage stats & other more details insights are only available to Pro & Enterprise customers
• There's a toggle that lets you slide between the cumulative and monthly view. If you're on the free tier and you press the toggle, a modal pops up letting you know it's a Pro feature and you'll need to upgrade.

[jmtaber129]

In general, I strongly believe that we should do our best to accurately and honestly represent usage data to our users, including free-tier users we're trying to convert. I think intentionally only showing a chart that always goes up and to the right to assert that current usage is good even when current usage is potentially declining is purposefully misleading, especially when we don't make it clear that the chart is showing cumulative data. IMO this would also be extremely transparent to a user base of mostly engineers + engineering leaders, which could easily damage our users' trust in both the Parabol product and the organization, ultimately doing more harm than good.

If we want to accurately represent meeting usage while also combatting conclusions of declining usage when monthly meeting counts go down, we could:

• Make sure we only show meetings with "activity" - i.e. don't include a retro with no reflections, don't include a check-in with only one participant, don't include sprint poker with no issues, etc.. This would allow us to show "true" meeting usage.
• Add a chart annotation or help tooltip that calls out that teams run less meetings during certain holiday periods
• Show other monthly stats that could be more closely tied to value - # of tasks generated from meetings, # of issues scored in poker, # of reflections in retros, etc.. These stats could support the argument that Parabol is helping your team become more efficient and collaborate better, even when total meeting counts are declining.
• # of meetings in past N days (e.g. 30 days) - even if meeting activity has declined, this would show that there's still active usage. We could apply this to other stats, too.

Regardless, if we want to show cumulative meeting counts for any reason, we'd need to make some changes to accurately represent this data in a way that's not misleading:

• Clearly label this as cumulative, and not broken down by month
• Use a chart type that implies cumulative and not monthly - i.e. switch from a bar chart (which implies monthly/period-bound stats) to a stacked line chart (which is more common for cumulative stats)

[nickoferrall]

For sure, we're not trying to mislead users, and I agree that we need to add a cumulative label to the chart.

I think cumulative data is still valuable. Other than the chart moving up and to the right, it's interesting to see that we've had more check-ins than retros and the total count of each meeting type.

A monthly meeting chart is more useful though as it's easier to see the changes, e.g. it'd be helpful to easily compare last summer's holiday season dip with this summer. I can see why you feel it's misleading to show the cumulative chart without the monthly chart, so I think we should show both in the free tier.

I do suspect that restricting some data insights to Pro/Enterprise users would be a helpful conversion tool. If monthly meetings doesn't feel right, then perhaps more detailed insights such as sentiment analysis on reflections would be a good place to start.

[avivapinchas]

purposefully misleading

These are pretty strong words, and I think unfair. What we've done is launch a first version of this feature, that we're turning on case-by-case for certain users to get feedback.

What seems 'obvious' to you or us may not be so to users. This first version is certainly imperfect, but it's made so we can get feedback on what is or isn't clear today. Should we hear from users that the cumulative chart is unclear, we would change it.

I do suspect that restricting some data insights to Pro/Enterprise users would be a helpful conversion tool.

I agree - larger orgs will have different needs around reporting and insights, and it would make sense to include those kinds of features in a paid tier because (1) they're valuable and (2) they just aren't relevant for smaller accounts using us.

But: I would considering 'insights' a separate feature, outside the scope of this effort. To me, Usage Stats is not about insights, helping managers make decisions or helping a team run better meetings - it's a simple tool so you can, at a basic level, understand how much your company is using Parabol.

[jmtaber129]

These are pretty strong words

This was intentional to emphasize the importance of the ethical challenges around product decisions related to visualizing data. For example, I would be uncomfortable with keeping this particular chart in its current state and then releasing it - not because it's an early version that may not be perfect, but because the current impact + potential biases behind it could have ethical implications that I'm not comfortable with.

I have no tensions with this chart being implemented like this in early versions, but as we make the present decision around whether to change this chart and in what ways, we should be aware of our inclinations to try to lead users to a particular conclusion with this data, and ensure that we're not allowing those biases to prompt a decision that could cause us to mislead users.

I do suspect that restricting some data insights to Pro/Enterprise users would be a helpful conversion tool

I definitely agree! The questions we want to answer for free users that we're trying to convert are probably very different from the questions that users may be willing to pay for. It would be totally reasonable for many usage visualizations to be premium features.

[avivapinchas]

we should be aware of our inclinations to try to lead users to a particular conclusion with this data, and ensure that we're not allowing those biases to prompt a decision that could cause us to mislead users.

Just speaking for myself: the intent here wasn't to create a Usage report that would be most likely to convert. We have an existing manual report we share with potential Enterprise users to show their usage, and we were mimicking the data shared there.

But I do agree that the distinction between the top section (which is Active) and the charts (which are cumulative) is unclear.

[mattkrick]

picture of a real org we're trying to convert

My proposal:

• rename Meetings to Total Meetings
• add tooltip icons next to the summary stat labels, e.g. "active teams". the tooltip should be human-friendly, not a developer spec.
• tooltip the chart titles if we can (I think I can? :steam_locomotive: :steam_locomotive: )

I don't like that active members/teams is 10% of total members/teams. Feels bad. Not sure how to solve for it

Up for other ideas! Just throwing these out to build upon

[avivapinchas]

I like those changes as a next step @mattkrick - we'll probably learn more as folks start to give feedback

[github-actions[bot]]

Stale issue