What is nonresponse bias?

Also called: Non-response bias, Participation bias

Nonresponse bias is the error that occurs when the people who don't answer a survey or join a study differ in meaningful ways from the people who do, so results describe the responders instead of the group you meant to study.

Updated

Unit and item nonresponse

  • Unit nonresponse: someone you invited doesn't take part at all. This is what most people mean by nonresponse bias.
  • Item nonresponse: someone takes part but skips particular questions. If the people who skip a pricing question are mostly on your biggest plans, the average you report for that question is off.

Response rate is not the same as bias

A low response rate raises the risk of nonresponse bias, but it doesn't prove you have it, and a high response rate doesn't rule it out. Robert Groves made this point in a 2006 review in Public Opinion Quarterly: what matters is whether the people who respond differ from those who don't on the thing you are measuring. A survey about export settings answered by a small share of customers can be fine if exporting habits have nothing to do with who replies. A churn survey can be badly biased even with a decent response rate.

What it looks like in product research

  • A churn survey is answered mostly by customers who left on reasonable terms. The angriest have stopped opening your emails.
  • An in-app survey only reaches people who log in, so satisfaction looks higher than it is across the whole customer base.
  • In B2B accounts, admins answer the feedback survey and the end users who work in the product every day don't.
  • Interview invites are accepted mostly by fans and power users, so the study hears little from people who barely use the product.

How to check for it

  • Compare responders with everyone you invited on things you already know: plan, tenure, usage level, region, role. Big differences are a warning.
  • Compare early and late responders. People who needed two or three reminders tend to look more like the people who never replied.
  • Follow up a small group of non-responders with one or two key questions by a different channel, and see whether their answers differ.
  • If responders are lopsided on a known attribute, weight the results or report each segment separately.

How to reduce it

Most of the fixes are the same ones that raise response rates: a short survey, a clear reason for asking, reminders, a sensible send time, a fair incentive and evidence that earlier feedback changed something. Aim them at the groups you are missing, not only at the overall rate.

Go deeper

Questions

What's the difference between nonresponse bias and sampling bias?

Sampling bias comes from who you choose to invite. Nonresponse bias comes from who chooses to answer once invited. A well-drawn sample can still end up biased if one segment ignores the invite.

What response rate is high enough to avoid nonresponse bias?

There is no safe threshold. Instead of chasing a number, check whether responders differ from non-responders on the attributes you know about, and on anything likely to relate to your question.

Do incentives fix nonresponse bias?

They raise response rates, which can help, but they can also attract a different kind of responder. Check the mix of responders after adding an incentive, the same way you would without one.

Next step

Run the study, not just the method

Great Question handles recruiting from your own customers, scheduling, incentives, interviews, tests and analysis in one place.

Book a demo

No credit card required.

Related terms

Full glossary
  • Sampling bias

    Sampling bias happens when the way you choose participants makes some kinds of people more likely to end up in your study than others, so the sample stops representing the users or customers you want to draw conclusions about.

  • Survey fatigue

    Survey fatigue is the drop in willingness or attention that sets in when people are asked to complete too many surveys, or surveys that are too long, so they stop responding, quit partway through or rush their answers.

  • Response bias

    Response bias is a systematic pull on what participants say, caused by question wording, who is asking, or how people want to come across, so their answers drift away from what they actually think or do.