What is response bias?
Also called: Survey response bias, Reporting bias
Response bias is a systematic pull on what participants say, caused by question wording, who is asking, or how people want to come across, so their answers drift away from what they actually think or do.
Updated
Response bias vs sampling and nonresponse bias
Response bias is about what people say once they are in your study. Sampling bias and nonresponse bias are about who ends up in it. A survey can reach a perfectly representative group of customers and still produce skewed answers. The fixes sit in different places: you reduce response bias through question wording and study setup, and you reduce the other two through recruiting and follow-up.
The main types
- Social desirability bias: people describe themselves the way they think they should be, for example overstating how often they review security settings.
- Acquiescence bias: people agree with statements, or say yes, regardless of what the statement says.
- Leading-question effects: wording such as "How much did you enjoy..." pushes answers toward the positive end.
- Question order effects: an earlier question sets up how people read a later one. Asking about a recent outage before overall satisfaction lowers the satisfaction score.
- Extreme or midpoint responding: some people habitually pick the ends of a scale, others always pick the middle.
- Demand effects: in usability tests, participants try to be helpful and say what they think the moderator wants to hear.
What it looks like in product research
- A usability participant struggles with a task for four minutes, then rates it "pretty easy".
- A concept survey asks how useful a planned feature would be, and nearly everyone picks "very useful". Usage after launch tells a different story.
- An account manager sends the satisfaction survey the day after resolving a big support ticket, to the customers they get along with best.
How to reduce it
- Ask about past behaviour instead of opinions or predictions: "When did you last export a report?" beats "Do you find exporting useful?"
- Use neutral wording and balanced scales, with as many negative options as positive ones.
- Put general questions before specific ones, and randomize option order where the list has no natural order.
- Separate the person asking from the thing being tested. A designer moderating a test of their own design invites politeness.
- Offer anonymity where you can, and say honestly how responses will be used.
- Check stated answers against what people do: product analytics, support tickets, or observed task success.
Go deeper
- Survey research guide
Covers the other survey biases that sit alongside response bias.
- How to write good survey questions
Bad questions rewritten so they don't push respondents toward an answer.
- How incentives affect bias in UX research
Whether paying participants changes what they tell you.
- Surveys in Great Question
Build and send surveys to your own customers.
Questions
What's the difference between response bias and nonresponse bias?
Response bias distorts the answers you get. Nonresponse bias distorts which answers you get, because the people who didn't respond differ from those who did. You can have either one without the other, so check for both.
Can you remove response bias completely?
No. Any question asked by a person or a company shapes the answer to some degree. The aim is to keep the pull small, keep it the same across the groups you compare, and confirm important findings with behavioural data.
Do incentives cause response bias?
When payment is framed as compensation for time, the main effect is on who signs up, which is a selection problem. Framing the incentive as a reward for positive feedback is what pushes answers. Keep the wording neutral and pay the same amount whatever people say.
Run the study, not just the method
Great Question handles recruiting from your own customers, scheduling, incentives, interviews, tests and analysis in one place.
No credit card required.
Related terms
Full glossary- Social desirability bias
Social desirability bias is the tendency for research participants to answer in ways that make them look good, overreporting behaviour they think is approved of and underreporting behaviour they think is frowned on.
- Acquiescence bias
Acquiescence bias is the tendency for survey respondents and interviewees to agree with statements or answer yes regardless of what is being asked, which inflates agreement on agree/disagree items and yes/no questions.
- Nonresponse bias
Nonresponse bias is the error that occurs when the people who don't answer a survey or join a study differ in meaningful ways from the people who do, so results describe the responders instead of the group you meant to study.