New question format: pro/anti likert

[SachaG]

Problem

With a normal likert scale, the survey designer has to pick a "pro" or "anti" formulation for each question, which risks

1. biasing the results
2. alienating part of the audience as they systematically have to check "disagree"

Example:

• pro: "immigration is net positive for a country"
• bad: "immigration is a net negative for a country"

Goal

The end goal would be the ability to visualize all responses on a left/right axis, to see at a glance a "blob" of how far along the axis the overall respondent set falls. But that requires a constant left/right scale for every question.

Idea 1

One idea could be to word each statement in a value-neutral way, such as "immigration: bad/neutral/good". But the issue becomes that the value-neutral formulations are often too vague to let respondents form an opinion. And the more details you provide, the more bias you risk introducing.

Idea 2

You could also just ask each question twice using both formulations, but that would make the surveys way too long and also be frustrating for respondents.

Idea 3

Or: each item should randomly switch between pro/anti formulations for each respondent, ensuring the number of pro/anti propositions is always balanced ~50/50.

Then, any "reversed" proposition should also reverse its scale (e.g. "strongly agree" counts the same as "strongly disagree" if the proposition wasn't reversed).

[LeaVerou]

Typically the way this is dealt with is making sure a certain point of view doesn't resort in all agrees/disagrees, which I believe is Idea 3.