Teaching · For Teachers

Two raters.
One stack of papers.

See how far they agree. Paste each rater’s scores, or each coder’s codes, one item per line. The page counts agreement, corrects it for chance, shows where the two part ways and lists the papers to put on the table first.

Open the calculator What you get

$12one-time · every device you sign in on

Buy · $12

Included with Caveat Classroom and Whole-school licenses. Or get it in the For Teachers toolkit, $79 for 72 items.

The agreement calculator

The worked example is made up: two teachers scoring 30 essays for use of evidence on a rubric of 1 to 4. Switch to Codes for two researchers coding 40 interview excerpts. Type over either, or start blank.

Loading the calculator. If nothing appears, turn on JavaScript.

What you get

Teaching · For Teachers$12 · included with Classroom and Whole-school licenses

What you finish with: For rubric scores: exact agreement, agreement within one level, Cohen’s kappa with an approximate 95 percent interval, linear and quadratic weighted kappa, each rater’s average and how each used the scale. For codes: percent agreement, Cohen’s kappa, Krippendorff’s alpha and agreement on each code. For both: the agreement matrix as a shaded grid, the items to discuss with the widest gaps first, and a record of saved sets so calibration rounds sit side by side. Prints a report and downloads the items, the matrix and the record as spreadsheets.
Who it is for: Teachers and grade-level or department teams calibrating a shared rubric, instructional coaches, assessment coordinators, and graduate students and researchers double-coding essays, observations or interview excerpts. Paper numbers or codes only; the page refuses a name.
Format: Browser tool. Type or paste the ratings and the results update as you type. Prints a report; downloads spreadsheets (.csv); export and import a file.
Access: $12 one-time, yours on every device you sign in on; the tool opens once you are signed in with it. Included with Caveat Classroom and Whole-school licenses. Also in the For Teachers toolkit ($79) and the Teaching Suite pass ($199), and 1 more bundle. Everything you type stays in this browser; nothing is sent to Caveat.
Where it lives: The For Teachers space in Caveat Teaching.
Support: Questions, a correction or a request: team@caveatacademic.com.

How to use it

Have two people score the same set of papers without conferring. Twenty to thirty papers that span the range of quality is a sound start.
Paste the results one item per line: a paper number or code, the first rater’s score, the second rater’s score. Two or three columns copied from a spreadsheet work as they are.
For rubric scores, type the lowest and highest level on the rubric so the matrix shows the whole scale, even when nobody gave a 1.
Read exact agreement and kappa together, then open To discuss. Put the widest gaps on the table with the rubric beside them and agree what each level means.
Save the set as round 1. Score a fresh batch after the conversation, save it as round 2, and compare the two under Saved.

Open the calculator

How the numbers are made

Agreement, the agreement chance would produce, and what is left.

The worked example: 21 of 30 essays got the same score from both raters. Chance alone would produce agreement of about 0.30 given how each rater used the scale, so kappa is 0.57.

Percent agreement and kappa

Exact agreement is the share of items both people rated the same. It is easy to read and easy to inflate: when most papers earn a 3, two raters who both give 3s to everything agree most of the time without judging anything.

Cohen’s kappa removes that. It takes the agreement expected if each rater kept their own mix of scores but assigned them at random, and reports how much of the remaining room the raters filled: 0 is chance, 1 is complete agreement, and a negative value is worse than chance. The 95 percent interval uses the large-sample standard error given by Cohen and repeated by McHugh; treat it as approximate, and wide when the set is small.

Scores: near misses count

Plain kappa counts a 3 against a 4 as a miss the same size as a 1 against a 4. For ordered rubric levels, weighted kappa counts each disagreement by its distance. With linear weights, two levels apart counts twice as much as one; with quadratic weights, four times as much, so near misses weigh less and wide ones more. Report which you used. The page also gives agreement within one level, which many scoring programs track beside exact agreement.

Typing the rubric’s lowest and highest level makes the matrix show every level, including one nobody used. It does not change the statistics.

Codes: alpha, and what the words mean

For unordered codes the page adds Krippendorff’s alpha for two coders and nominal data, the usual companion to kappa in content analysis. The two usually land close together. They differ in how chance agreement is figured: alpha pools both coders’ use of the codes and corrects for the size of the set. The build checks the alpha routine against the two worked examples Krippendorff published.

Landis and Koch offered words for bands of kappa (slight, fair, moderate, substantial, almost perfect) and called the divisions arbitrary themselves. McHugh argued for a stricter reading where decisions rest on the ratings. The page shows both and leaves the standard to you.

Sources

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37–46.
Cohen, J. (1968). Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychological Bulletin, 70(4), 213–220.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. The bands of kappa named on this page.
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. The stricter reading of kappa and the standard error used for the interval.
Krippendorff, K. (2011). Computing Krippendorff’s alpha-reliability. University of Pennsylvania, Annenberg School for Communication. Examples A and B (alpha of 0.095 and 0.692) are the test cases for the alpha routine here.
Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549.
Stemler, S. E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research, and Evaluation, 9(4).

Both worked examples are made up, including the item codes and the scores. The statistics describe how two people rated one set of items; they do not say which rater is right, and they are not a judgment of a teacher or a student. Ratings stay in this browser. Support: team@caveatacademic.com.

Two raters.One stack of papers.