Devographics
Diversity & resources selection
A previous thread (https://github.com/Devographics/surveys/issues/52) was opened about using more objective metrics to select resources to feature as pre-written, selectable options when people are taking a survey.
But relying solely on metrics presents the issue that we reinforce existing biases. For example, if mainly English speakers take the survey, then English-speaking video creators will get the most votes, and appear at the top of the rankings year after year even if some Spanish streamers may have more total viewers. And the same problem exists not just with language but with gender, race, etc.
Some potential solutions I'm considering:
1) Keep the rankings value-neutral but try to address the problem at the root by diversifying the survey audience.
- Pros: cleanest solution
- Cons: really hard to do, might not ensure a diverse final list
2) Manually set aside spots in each list for a quota of gender/race/language minorities
- Pros: ensures diversity
- Cons: hard to figure out right quota or where items should be seeded in the list
3) Set aside spots in each list for answers by gender/race/language minorities
In other words, instead of featuring e.g. women video creators, we'd feature video creators cited by women respondents (which may or may not be women themselves).
- Pros: less subjective than 2)
- Cons: might not ensure a diverse final list; same practical seeding issues as 2)
Current Thoughts
I currently think 2) might be the best solution, if done right.
For example, out of a list of 10 items the first 7 could be ranked according to an objective metrics, while the last 3 would be hand-picked to highlight minoritized demographics (or even just less visible new entrants in a category that deserve to be highlighted), along with some kind of visual tag to indicate that.
Obviously this will irk some people (the "leave your politics out of my code!" crowd) but I think erring on the side of transparency is good, and clarifying why items appear or not in the survey it something we should be doing a much better job of anyway.
(Note that this is kind of what I'm already doing more or less by default, but this would clarify the process)
Btw it's interesting to note that pre-defined questions get about 10x the number of responses compared to similar freeform questions (using the "Video Creators" vs "People You Follow" questions as basis).
This alone is a good reason to try and figure out a good set of pre-defined options instead of relying too much on freeform input.
CSS 2023 update: predefined options now appear in a randomized order, which at least solves the problem of knowing where to put items in each list.
Shaine here. @atopal mentioned in the last meeting that we've got no frame to use for sampling means we cannot do probability sampling.
As a solution, we can redefine our population by limiting it to some group in which we can get a complete list of members so we can do random sampling and generalize findings to that group. This takes more time and effort given the situation. I'm also guessing it could be hard to get that list and to initially redefine the population given the survey goals.
Suggestions
Non-probability sampling techniques such as what was applied in the past years can be used. Results from this, however, should be interpreted with caution. Data from non-probability sampling techniques cannot be used to generalize results to the entire population. The good news is, we can still do descriptive analysis and doing this will still be aligned to the survey's goal and data usage as indicated here (particularly the italicized ones as shown below).
What is the survey's goal? The survey's goal is to track the evolution of upcoming features and libraries, and help developers decide which new technologies to focus on.
How will this data be used? All data collected will be released openly for any developer or company to consult. Browser vendors also use this data to prioritize focus areas and inform their roadmaps.
The question now is: How can we can collect the data that we need to allow us to perform the analysis method that is assumed to address the survey goals, given the target launch date?
We can do the following:
- get as many target respondents as possible by making sure we make the survey available to them, esp. to the subgroups we are interested about
- anybody who writes CSS, whether regularly or occasionally, as part of their job, as a student, or just for fun as indicated here
- respondents who have hopefully, 100% completion rate so that we can
- obtain data that would allow us to describe the patterns in the results in context (data on demographics will be valuable on this). This means, we get results that are not generalizable to the population of people who write CSS but we are capable of describing the diversity of choices within the group of respondents (descriptive analysis)
- through clustering or
- simple correlations in plots made possible by demographic data
To add, given the past results we need to make sure we make the questionnaire available to communities of female CSS coders, and other subgroups of interest from which we got low number of respondents. @SachaG, @atopal let me know your thoughts!
For questions that have options, we have this “primacy” effect in which respondents tend to choose items at the top of the list in self-administered surveys (see reference).
Randomly arranged options attain its purpose iff for each respondent, a different order is applied. However, if that randomized order is the same for all the respondents, it will not serve its purpose.
Because of concerns about the effects of category order on responses to closed-ended questions, many sets of response options in Pew Research Center’s surveys are programmed to be randomized to ensure that the options are not asked in the same order for each respondent. Rotating or randomizing means that questions or items in a list are not asked in the same order to each respondent (see reference).
For sets of choices that have a natural order, we can use it (ex: sad < neutral < happy or happy > neutral > sad).
In the absence of natural order, we can organize options according to popularity based from some pilot test results or recent researches. This will however require us some effort to attain. Saw a discussion on that here https://github.com/Devographics/surveys/issues/52. In this way, options that are more likely to be chosen are presented first, lessening the effort it takes for majority of the respondents to go through the entire list. This also makes it less likely for primacy effect to produce unusual, unexpected results. @michaelquiapos, you might have an idea if it is also okay to use results from previous survey (State of CSS 2022) to arrange options based on popularity especially those that present a long list.
An alternative and easy option, w/o data on popularity, would be to alphabetically arrange the options for easy to follow logic. If respondent has an answer in mind which he or she wants to tick, it will be easy to search through the list. This does not however, help address the primacy effect.
I actually opened a new thread specifically about this issue, maybe we can continue the discussion there? https://github.com/Devographics/surveys/issues/92