What is sampling bias?

Also called: Sample bias

Sampling bias happens when the way you choose participants makes some kinds of people more likely to end up in your study than others, so the sample stops representing the users or customers you want to draw conclusions about.

Updated

The classic example

In 1936 the Literary Digest magazine mailed straw-poll ballots to millions of people drawn largely from its own subscribers, telephone directories and car registration lists, and predicted that Alf Landon would beat Franklin Roosevelt. Roosevelt won by a landslide. During the Depression, people with phones and cars were better off than average, and the sample leaned their way. Later analyses also point to who bothered to mail ballots back, a nonresponse problem. Huge numbers didn't save the poll, because the sample was skewed before anyone answered.

Where it creeps into product research

  • Recruiting from your most engaged customers: the beta list, the community forum, the people who answered last time.
  • Account managers nominating participants, and picking the customers they get along with.
  • Open calls on social media, which reach followers and fans.
  • In-app intercepts, which only reach people who are active right now.
  • Leaving out churned customers, lost deals and people who tried the product once, so you only hear from those it already works for.
  • Drawing on the same small group of panel members study after study.

Sampling bias in qualitative research

Interviews and usability tests don't aim for a statistically representative sample. You choose people on purpose, for example a mix of new and long-standing admins. That deliberate choice is purposive sampling. Sampling bias is when the mix is shaped by accident, by whoever was easiest to reach, and the study then claims to speak for everyone.

How to reduce it

  • Define the population before recruiting: which users, plans, roles and markets the findings should apply to.
  • Build the invite list from your CRM or product data, so you can see who is in it and who isn't.
  • Set quotas by segment, such as usage level or company size, and screen on behaviour instead of self-description.
  • Track how often each person takes part and give priority to people who haven't been in a study recently.
  • Recruit churned customers and non-users on purpose when the question concerns them.
  • Report who you didn't reach, so readers know the limits of the findings.

Go deeper

Questions

Does a bigger sample fix sampling bias?

No. A larger sample reduces random error, but if the selection method favours one group, you just get a more confident wrong answer. The Literary Digest poll is the standard example.

Is convenience sampling always biased?

It is always at risk, because the people easiest to reach are rarely typical. For early exploratory interviews that can be acceptable, as long as you say who you spoke to and don't generalize beyond them.

What's the difference between sampling bias and selection bias?

The terms overlap and are often used interchangeably. Selection bias is the broader idea: any process that decides who ends up in the data in a skewed way, including who drops out. Sampling bias usually refers to how the sample was drawn in the first place.

Next step

Run the study, not just the method

Great Question handles recruiting from your own customers, scheduling, incentives, interviews, tests and analysis in one place.

Book a demo

No credit card required.

Related terms

Full glossary
  • Convenience sampling

    Convenience sampling is a non-probability sampling method where you recruit whoever is easiest to reach, such as customers who answer your first email or colleagues down the hall, instead of choosing participants to represent a defined population.

  • Quota sampling

    Quota sampling is a non-probability sampling method where you set a target number of participants for each group that matters, such as plan tier or role, and recruit until each quota is full, so the sample's mix matches the population you care about.

  • Nonresponse bias

    Nonresponse bias is the error that occurs when the people who don't answer a survey or join a study differ in meaningful ways from the people who do, so results describe the responders instead of the group you meant to study.