Teaching · For Teachers

Find the question
that tested nothing.

Paste the answer key and your students' answers. Every item gets its difficulty, how well it separates strong and weak scorers, and a look at which wrong answers drew whom, so you can tell a hard question from a broken one.

Open the calculatorWhat you get
$12one-time · every device you sign in on
Buy · $12

Included with Caveat Classroom and Whole-school licenses. Or get it in the For Teachers toolkit, $79 for 72 items.

The calculator

The worked example is a made-up 20-item unit test taken by 30 students, coded T-01 to T-30, with one item whose key was entered wrong. Paste your own answers on the Answers tab.

Loading the calculator. If nothing appears, turn on JavaScript.

What you get

Teaching · For Teachers$12 · included with Classroom and Whole-school licenses

What you finish with
For every item: the difficulty index (proportion correct), the upper-lower discrimination index, the corrected item-total correlation, a chart of each option chosen by the upper and lower groups and a plain flag (keep, review, revise or check the key). For the whole test: mean, standard deviation, KR-20 reliability, the standard error of measurement and the score distribution. Prints a report with no student codes and downloads the table as a spreadsheet.
Who it is for
Classroom teachers, department teams and instructional coaches reviewing their own multiple-choice tests and quizzes, and education students learning classical item analysis.
Format
Browser tool. Reads pasted answers or a .csv export from a scanner or spreadsheet; prints a report; downloads the item table as a spreadsheet (.csv); export and import a file.
Access
$12 one-time, yours on every device you sign in on; the tool opens once you are signed in with it. Included with Caveat Classroom and Whole-school licenses. Also in the For Teachers toolkit ($79) and the Teaching Suite pass ($199), and 1 more bundle. Everything you type stays in this browser; nothing is sent to Caveat.
Where it lives
The For Teachers space in Caveat Teaching.
Support
Questions, a correction or a request: team@caveatacademic.com.

How to use it

  1. Enter the answer key, one letter per item, and paste one line per student: a code, a comma, then the answers. Scored 1 and 0 data works too.
  2. Fix any line the page flags, such as the wrong number of answers or a name where a code belongs, then press Analyze.
  3. On the Items tab, sort by difficulty or discrimination and open any item to see which options the upper and lower groups chose.
  4. Check the key on anything flagged that way, revise items that do not discriminate, and replace options no one chose.
  5. Read the Whole test tab for reliability and the score spread, then print the report for your team.

Open the calculator

How the statistics are worked out

Classical test theory, with the formulas the page uses.

Item statistics

The difficulty index p is the proportion of students who answered the item correctly, so a higher p means an easier item. The discrimination index D is p in the upper group minus p in the lower group, where the groups are the top and bottom 27 percent of students by total score (Kelley's 27 percent; you can choose 25, 33 or 50). Ties at the group boundary go by the order the students were entered.

The item-rest correlation r is the Pearson (point-biserial) correlation between the item's 0 or 1 score and each student's total on the other items, so the item does not inflate its own correlation. Blanks count as wrong.

Flags and reliability

The flags follow Ebel and Frisbie's guidance for classroom tests: D of .40 or more is very good, .30 to .39 good, .20 to .29 worth a review, under .20 worth revising, and a negative D or r usually means a wrong key or two defensible answers. An option chosen by more upper-group students than the key is flagged as a possible wrong key; an option chosen by fewer than 5 percent of students is not doing its job as a distractor.

KR-20 = k ÷ (k − 1) × (1 − Σpq ÷ variance of total scores), with the population variance. The standard error of measurement is the standard deviation × √(1 − KR-20).

Item statistics from one class are rough. With 30 students, a D of .20 can move by .15 or more from one group to the next. Read the item before changing it.

Sources

  1. Ebel, R. L., and Frisbie, D. A. (1991). Essentials of Educational Measurement (5th ed.). Prentice Hall. Discrimination index guidance for classroom tests.
  2. Kelley, T. L. (1939). The selection of upper and lower groups for the validation of test items. Journal of Educational Psychology, 30(1), 17–24. The 27 percent groups.
  3. Kuder, G. F., and Richardson, M. W. (1937). The theory of the estimation of test reliability. Psychometrika, 2(3), 151–160. KR-20.
  4. Crocker, L., and Algina, J. (2008). Introduction to Classical and Modern Test Theory. Cengage. Item difficulty, the corrected item-total correlation and the standard error of measurement.
  5. Haladyna, T. M., and Rodriguez, M. C. (2013). Developing and Validating Test Items. Routledge. Distractor analysis and nonfunctioning distractors.

The worked example is made up and generated by a model (30 students, 20 items; KR-20 0.86 in this sample). Keep only student codes here; the printed report shows no codes. Support: team@caveatacademic.com.