Teaching · For Teachers

Does the score predict what it claims to?

Enter each outcome the measure was correlated with and, for a screener, who was flagged and who later struggled. The organizer works out the intervals and the classification figures, and says where the evidence is thin.

Start an evidence recordWhat you get
$12one-time · every device you sign in on
Buy · $12

Included with Caveat Classroom and Whole-school licenses. Or get it in the For Teachers toolkit, $79 for 84 items.

The evidence organizer

One record per measure and grade. The example screener and its numbers are made up.

Loading the form. If nothing appears, turn on JavaScript.

What you get

Teaching · For Teachers$12 · included with Classroom and Whole-school licenses

What you finish with
A one-page summary of the criterion evidence for one measure: each correlation with its interval and size, classification accuracy at the cut score, and what is missing.
Who it is for
Teachers, assessment coordinators, school psychologists and school-based researchers choosing or checking a measure.
Format
Browser organizer. One saved record per measure; printed with its summary or as a blank form; CSV of the correlation rows; export and import a file.
Access
$12 one-time, yours on every device you sign in on; the tool opens once you are signed in with it. Included with Caveat Classroom and Whole-school licenses. Also in the For Teachers toolkit ($79) and the Teaching Suite pass ($199), and 1 more bundle. Everything you type stays in this browser; nothing is sent to Caveat.
Where it lives
The For Teachers space in Caveat Teaching.
Support
Questions about an order, access or a correction: team@caveatacademic.com.
fall scorespring41309132does the score predict what comes next
Drawn for this page. A correlation shows how well scores line up with an outcome across a group; the two-by-two counts show how many students a cut score catches and misses.

How to use it

  1. Write down the decision the scores are used for: that decides which evidence matters.
  2. Add correlations from your own data first, then from the manual.
  3. For a screener, count flagged and not flagged students against the later outcome.
  4. Answer the quality checks honestly; a no is useful information.
  5. Redo the counts each year with the new cohort.

Start an evidence record

What the organizer works out

Concurrent and predictive

Concurrent evidence correlates the measure with a criterion taken at about the same time. Predictive evidence uses an outcome measured later. A measure used to predict needs predictive evidence.

The interval matters

A correlation from 38 students can sit anywhere from about .20 to .70. The organizer gives a 95% interval with the Fisher z method so a small study does not look stronger than it is.

What it is not

Validity belongs to a use of scores, not to the test. This record gathers one kind of evidence, relations to a criterion; content and fairness evidence still count.

Sources

  1. American Educational Research Association, American Psychological Association and National Council on Measurement in Education. Standards for Educational and Psychological Testing, 2014, chapter 1 (evidence based on relations to other variables).
  2. Cohen, J. Statistical Power Analysis for the Behavioral Sciences, 2nd ed., 1988 (conventions of .10, .30 and .50 for small, medium and large correlations).
  3. Fisher, R. A. On the probable error of a coefficient of correlation deduced from a small sample. Metron, 1921 (the z transformation used for the interval).
  4. Jenkins, J. R., Hudson, R. F. and Johnson, E. S. Screening for at-risk readers in a response to intervention framework. School Psychology Review, 2007.
  5. National Center on Intensive Intervention. Academic screening tools chart: classification accuracy and validity standards.

The example measure, students and figures are made up. Support: team@caveatacademic.com.