Devographics
State of JS 2024 sentiment issue
TL;DR: this year we started giving people the option to remain neutral instead of picking positive/negative, and as a result positive sentiment went way down.
Here's the JS 2024 "front end positivity" chart as it currently stands (survey still ongoing as of now):
The drop is almost certainly due not to a change in respondents sentiment, but to the way we calculate positivity.
Pre-2024, we used a 5-option "scale" to collect data:
- Never heard
- Heard about it, not interested
- Heard about it, interested
- Used it, want to use it again
- Used it, do not want to use it again
Starting this year, we switched to this 3-option + sentiment scale:
- Never heard [posititve/negative/neutral]
- Heard about it [posititve/negative/neutral]
- Used it [posititve/negative/neutral]
In order to harmonize both scales, I decided to code the legacy 5-option scale as follows:
- Never heard -> never heard/neutral
- Heard about it, not interested -> heard about it/negative
- Heard about it, interested -> heard about it/positive
- Used it, want to use it again -> used it/positive
- Used it, do not want to use it again -> used it/negative
This works, but there is one issue: neutral values for heard/used don't exist in the legacy data. This means that 100% of "used it" respondents are allocated between positive and negative buckets, resulting in e.g. a 80/20 positive/negative split for library Foo.
When the neutral value does exist though (for 2024) the split is much more likely to be something along the lines of 60/30/10 positive/neutral/negative for the exact same library since many people will default to "neutral".
So the result is this impression that Foo's positivity dropped from 80% to 60% year-to-year.
Solution
I think I need to change how positivity/negativity are calculated so that neutral responses are discarded, in order to harmonize both scales. Hopefully this doesn't have any other unforeseen consequences…
The main issue is that this chart for example is built around the idea of showing the positive/neutral/negative breakdown:
So we essentially would introduce two similar but distinct "positivity" ratios:
- The "absolute" positivity ratio of 51.4%
- The "relative" positivity ratio of 93% (because 93% of people who expressed a sentiment had a positive sentiment)
That's probably ok (and in fact the relative ratio seems more meaningful) but we'll just have to be careful about not making things too confusing…
(Thinking some more about it, maybe that chart could just default to hiding that middle "neutral" grouping altogether, which would simplify it visually as a bonus)