What is nonresponse bias?
Also called: Non-response bias, Participation bias
Nonresponse bias is the error that occurs when the people who don't answer a survey or join a study differ in meaningful ways from the people who do, so results describe the responders instead of the group you meant to study.
Updated
Unit and item nonresponse
- Unit nonresponse: someone you invited doesn't take part at all. This is what most people mean by nonresponse bias.
- Item nonresponse: someone takes part but skips particular questions. If the people who skip a pricing question are mostly on your biggest plans, the average you report for that question is off.
Response rate is not the same as bias
A low response rate raises the risk of nonresponse bias, but it doesn't prove you have it, and a high response rate doesn't rule it out. Robert Groves made this point in a 2006 review in Public Opinion Quarterly: what matters is whether the people who respond differ from those who don't on the thing you are measuring. A survey about export settings answered by a small share of customers can be fine if exporting habits have nothing to do with who replies. A churn survey can be badly biased even with a decent response rate.
What it looks like in product research
- A churn survey is answered mostly by customers who left on reasonable terms. The angriest have stopped opening your emails.
- An in-app survey only reaches people who log in, so satisfaction looks higher than it is across the whole customer base.
- In B2B accounts, admins answer the feedback survey and the end users who work in the product every day don't.
- Interview invites are accepted mostly by fans and power users, so the study hears little from people who barely use the product.
How to check for it
- Compare responders with everyone you invited on things you already know: plan, tenure, usage level, region, role. Big differences are a warning.
- Compare early and late responders. People who needed two or three reminders tend to look more like the people who never replied.
- Follow up a small group of non-responders with one or two key questions by a different channel, and see whether their answers differ.
- If responders are lopsided on a known attribute, weight the results or report each segment separately.
How to reduce it
Most of the fixes are the same ones that raise response rates: a short survey, a clear reason for asking, reminders, a sensible send time, a fair incentive and evidence that earlier feedback changed something. Aim them at the groups you are missing, not only at the overall rate.
Go deeper
- How to increase survey response rates
Practical ways to get more of the people you invite to answer.
- Survey research guide
Nonresponse alongside the other survey biases.
- How to analyze survey data
Segmenting and weighting results before you report them.
- Churn survey template
A survey type especially exposed to nonresponse bias.
Questions
What's the difference between nonresponse bias and sampling bias?
Sampling bias comes from who you choose to invite. Nonresponse bias comes from who chooses to answer once invited. A well-drawn sample can still end up biased if one segment ignores the invite.
What response rate is high enough to avoid nonresponse bias?
There is no safe threshold. Instead of chasing a number, check whether responders differ from non-responders on the attributes you know about, and on anything likely to relate to your question.
Do incentives fix nonresponse bias?
They raise response rates, which can help, but they can also attract a different kind of responder. Check the mix of responders after adding an incentive, the same way you would without one.
Run the study, not just the method
Great Question handles recruiting from your own customers, scheduling, incentives, interviews, tests and analysis in one place.
No credit card required.
Related terms
Full glossary- Sampling bias
Sampling bias happens when the way you choose participants makes some kinds of people more likely to end up in your study than others, so the sample stops representing the users or customers you want to draw conclusions about.
- Survey fatigue
Survey fatigue is the drop in willingness or attention that sets in when people are asked to complete too many surveys, or surveys that are too long, so they stop responding, quit partway through or rush their answers.
- Response bias
Response bias is a systematic pull on what participants say, caused by question wording, who is asking, or how people want to come across, so their answers drift away from what they actually think or do.