Teaching · For Teachers

What else could explain the result, checked before you claim it.

Choose your design, enter who started and who finished, and mark each classic threat as addressed or a risk. The summary works out attrition, gathers the open risks and names what your design cannot rule out.

Start a validity reviewWhat you get
$5one-time · every device you sign in on
Buy · $5

Included with Caveat Classroom and Whole-school licenses. Or get it in the For Teachers toolkit, $79 for 72 items.

The validity review

One record per study and review. No participant data are entered. The example is made up.

Loading the form. If nothing appears, turn on JavaScript.

What you get

Teaching · For Teachers$5 · included with Classroom and Whole-school licenses

What you finish with
For each study, the design, attrition overall and by group, every threat marked, the open risks grouped by kind, and notes on what the design leaves open.
Who it is for
Teachers doing action research, graduate students, program evaluators and district research staff planning or reporting a study.
Format
Browser checklist. Several saved reviews; printed with the summary or as a blank checklist; CSV with one row per review; export and import a file.
Access
$5 one-time, yours on every device you sign in on; the tool opens once you are signed in with it. Included with Caveat Classroom and Whole-school licenses. Also in the For Teachers toolkit ($79) and the Teaching Suite pass ($199), and 1 more bundle. Everything you type stays in this browser; nothing is sent to Caveat.
Where it lives
The For Teachers space in Caveat Teaching.
Support
Questions about an order, access or a correction: team@caveatacademic.com.
programresulttimewho joinedother events
Drawn for this page. A validity review asks what else, besides the program, could have produced the result.

How to use it

  1. Start a record with a study code and the date.
  2. Choose the design and write the question.
  3. Enter the numbers enrolled and with outcome data for each group.
  4. Mark each threat as addressed, a risk or not applicable.
  5. Write how the main risks will be handled or reported, and read the design notes.

Start a validity review

What the review is for

The classic threats

Internal validity asks whether the program, and not something else, caused the change: other events, normal growth, practice on the test, a changed measure, regression of extreme scorers toward the average, groups that differed at the start, or a comparison group that got some of the program. Construct, statistical and external validity ask whether you measured the right thing, had enough data, and can generalize.

Attrition

Losing students between the start and the outcome can bias results, especially when the groups lose students at different rates. Evidence reviews such as the What Works Clearinghouse look at both overall and differential attrition together; report both.

What this checklist is not

It does not rate a study against an evidence standard or compute statistical power. It organizes a design review.

Sources

  1. Shadish, W. R., Cook, T. D., and Campbell, D. T. Experimental and Quasi-Experimental Designs for Generalized Causal Inference (2002).
  2. What Works Clearinghouse, Institute of Education Sciences. Procedures and Standards Handbook, Version 5.0 (2022), attrition standards.
  3. Campbell, D. T., and Stanley, J. C. Experimental and Quasi-Experimental Designs for Research (1963).
  4. Kratochwill, T. R., and others (2010). Single-Case Designs Technical Documentation. What Works Clearinghouse.

The example study is made up. The checklist organizes a design review; it does not rate the study. Support: team@caveatacademic.com.