What is response bias?

Also called: Survey response bias, Reporting bias

Response bias is a systematic pull on what participants say, caused by question wording, who is asking, or how people want to come across, so their answers drift away from what they actually think or do.

Updated

Response bias vs sampling and nonresponse bias

Response bias is about what people say once they are in your study. Sampling bias and nonresponse bias are about who ends up in it. A survey can reach a perfectly representative group of customers and still produce skewed answers. The fixes sit in different places: you reduce response bias through question wording and study setup, and you reduce the other two through recruiting and follow-up.

The main types

  • Social desirability bias: people describe themselves the way they think they should be, for example overstating how often they review security settings.
  • Acquiescence bias: people agree with statements, or say yes, regardless of what the statement says.
  • Leading-question effects: wording such as "How much did you enjoy..." pushes answers toward the positive end.
  • Question order effects: an earlier question sets up how people read a later one. Asking about a recent outage before overall satisfaction lowers the satisfaction score.
  • Extreme or midpoint responding: some people habitually pick the ends of a scale, others always pick the middle.
  • Demand effects: in usability tests, participants try to be helpful and say what they think the moderator wants to hear.

What it looks like in product research

  • A usability participant struggles with a task for four minutes, then rates it "pretty easy".
  • A concept survey asks how useful a planned feature would be, and nearly everyone picks "very useful". Usage after launch tells a different story.
  • An account manager sends the satisfaction survey the day after resolving a big support ticket, to the customers they get along with best.

How to reduce it

  • Ask about past behaviour instead of opinions or predictions: "When did you last export a report?" beats "Do you find exporting useful?"
  • Use neutral wording and balanced scales, with as many negative options as positive ones.
  • Put general questions before specific ones, and randomize option order where the list has no natural order.
  • Separate the person asking from the thing being tested. A designer moderating a test of their own design invites politeness.
  • Offer anonymity where you can, and say honestly how responses will be used.
  • Check stated answers against what people do: product analytics, support tickets, or observed task success.

Go deeper

Questions

What's the difference between response bias and nonresponse bias?

Response bias distorts the answers you get. Nonresponse bias distorts which answers you get, because the people who didn't respond differ from those who did. You can have either one without the other, so check for both.

Can you remove response bias completely?

No. Any question asked by a person or a company shapes the answer to some degree. The aim is to keep the pull small, keep it the same across the groups you compare, and confirm important findings with behavioural data.

Do incentives cause response bias?

When payment is framed as compensation for time, the main effect is on who signs up, which is a selection problem. Framing the incentive as a reward for positive feedback is what pushes answers. Keep the wording neutral and pay the same amount whatever people say.

Next step

Run the study, not just the method

Great Question handles recruiting from your own customers, scheduling, incentives, interviews, tests and analysis in one place.

Book a demo

No credit card required.

Related terms

Full glossary
  • Social desirability bias

    Social desirability bias is the tendency for research participants to answer in ways that make them look good, overreporting behaviour they think is approved of and underreporting behaviour they think is frowned on.

  • Acquiescence bias

    Acquiescence bias is the tendency for survey respondents and interviewees to agree with statements or answer yes regardless of what is being asked, which inflates agreement on agree/disagree items and yes/no questions.

  • Nonresponse bias

    Nonresponse bias is the error that occurs when the people who don't answer a survey or join a study differ in meaningful ways from the people who do, so results describe the responders instead of the group you meant to study.