What is inter-rater reliability?
Also called: Inter-coder reliability, Inter-rater agreement, Cohen's kappa
Inter-rater reliability is the degree to which two or more researchers, coding or rating the same data independently, reach the same answers. It tells you whether a codebook or rating scale produces consistent results, or depends on who happens to apply it.
Updated
Why product and UX teams care about it
When two researchers tag the same interview excerpts, they rarely agree on every tag. If one researcher codes 40% of excerpts as "pricing concern" and a colleague codes 15%, the finding you report depends on who did the coding. Measuring agreement on a sample of your data shows whether your code definitions are clear enough to share, and where they need tightening before the rest of the team uses them.
Most product teams don't need a reliability statistic for every study. It earns its place when a codebook will be reused across studies, when findings feed a big decision, or when you're training new people to tag research data.
Cohen's kappa, with a worked example
Raw percent agreement overstates reliability, because two coders will agree some of the time by chance. Cohen's kappa, introduced by Jacob Cohen in 1960, corrects for that. The formula is kappa = (observed agreement − chance agreement) ÷ (1 − chance agreement).
Say two researchers each read the same 50 interview excerpts and mark whether each one is a "pricing concern":
- Both say yes: 20. Both say no: 15. Researcher A yes, B no: 5. Researcher A no, B yes: 10.
- Observed agreement: (20 + 15) ÷ 50 = 0.70, so they agree 70% of the time.
- A says yes to 25 of 50 (0.5). B says yes to 30 of 50 (0.6). Chance agreement: (0.5 × 0.6) + (0.5 × 0.4) = 0.50.
- Kappa: (0.70 − 0.50) ÷ (1 − 0.50) = 0.40.
How to read the score
The bands most people use come from Landis and Koch (1977): below 0 is poor, 0.00–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial and 0.81–1.00 almost perfect. In the example above, 70% agreement sounds solid, but a kappa of 0.40 is only fair. The fix is usually a clearer code definition with examples of what does and doesn't count, then a second round on fresh excerpts.
Cohen's kappa works for two coders. For three or more coders, Fleiss' kappa or Krippendorff's alpha are the usual choices.
How to run a reliability check
- Write the codebook first, with a definition and one example for each code.
- Have two people code the same 10–20% of the data independently, without discussing it.
- Calculate agreement code by code. A low score on one code points to a definition problem, not a bad coder.
- Discuss the disagreements, revise the definitions, then code the rest.
Go deeper
- How to analyze interview data
Building the codebook that a reliability check tests.
- Qualitative data analysis guide
Coding approaches for turning transcripts into themes.
- What UX researchers get wrong about AI
How human coding agreement compares with AI-assisted analysis.
- Synthesis in Great Question
Tag and compare themes across interviews in one place.
Questions
What is a good inter-rater reliability score?
On the Landis and Koch bands, a kappa above 0.60 counts as substantial agreement. There's no universal pass mark, so agree a target with your team before coding starts, and when you miss it, look at which codes are pulling the score down.
What's the difference between inter-rater and intra-rater reliability?
Inter-rater reliability compares different people coding the same data. Intra-rater reliability compares one person coding the same data at two points in time, which shows whether their own judgement drifts.
Why not just report percent agreement?
Because chance inflates it. If a code applies to 90% of excerpts, two coders who say yes to almost everything will agree most of the time without reading carefully. Kappa subtracts that chance agreement, so it gives a fairer picture.
Run the study, not just the method
Great Question handles recruiting from your own customers, scheduling, incentives, interviews, tests and analysis in one place.
No credit card required.
Related terms
Full glossary- Triangulation
Triangulation is checking a research finding against more than one source, method or researcher, so you only act on it when different kinds of evidence point the same way.
- Member checking
Member checking is a qualitative research technique where you take your interpretation of what participants said back to them, so they can confirm, correct or add to it before you treat it as a finding.
- Atomic research
Atomic research is a way of storing research findings as small, linked units, such as a single observation tied to its source, instead of whole reports, so the evidence can be found, reused and combined across studies.