Teaching · For Teachers

Cleared to collect only after the codes agree.

Track each collector through the training steps, then score practice rounds against a master coder until agreement clears your cutoff. Keep checking for drift once collection starts.

Start a collector recordWhat you get
$5one-time · every device you sign in on
Buy · $5

Included with Caveat Classroom and Whole-school licenses. Or get it in the For Teachers toolkit, $79 for 72 items.

The collector record

One record per collector, by code. The example is made up.

Loading the form. If nothing appears, turn on JavaScript.

What you get

Teaching · For Teachers$5 · included with Classroom and Whole-school licenses

What you finish with
For each collector, the training steps, every practice round’s agreement against the cutoff, drift checks, and a clear cleared or not yet.
Who it is for
Research team leads, teacher researchers, program evaluators and district assessment coordinators who train observers, test administrators or interviewers.
Format
Browser form. Several saved records, one per collector; printed with the summary or as a blank training log; CSV of the practice rounds; export and import a file.
Access
$5 one-time, yours on every device you sign in on; the tool opens once you are signed in with it. Included with Caveat Classroom and Whole-school licenses. Also in the For Teachers toolkit ($79) and the Teaching Suite pass ($199), and 1 more bundle. Everything you type stays in this browser; nothing is sent to Caveat.
Where it lives
The For Teachers space in Caveat Teaching.
Support
Questions about an order, access or a correction: team@caveatacademic.com.
same case, same codes
Drawn for this page. Agreement is checked by having the collector and a master coder score the same cases and counting where they match.

How to use it

  1. Start a record with a collector code and the training start date.
  2. Choose the role and name the instrument.
  3. Mark each training step as it is done.
  4. Score practice cases alongside the master coder and enter each round.
  5. Clear the collector once training is done and agreement meets the cutoff, then plan drift checks.

Start a collector record

What the record is for

Why agreement matters

Data are only as consistent as the people collecting them. Scoring the same practice cases as a master coder shows whether a collector applies the codes the way the protocol intends. Many studies require at least 80% exact agreement before collection, and some use a chance-corrected statistic such as Cohen’s kappa as well.

Drift

Over weeks of collection, people slowly drift from the codebook. Short agreement checks during collection catch it, and retraining on the items that slipped brings it back.

What this checklist is not

It computes percent agreement only. For kappa or other reliability statistics, use a reliability calculator. Your protocol sets the steps and the cutoff.

Sources

  1. Hartmann, D. P. (1977). Considerations in the choice of interobserver reliability estimates. Journal of Applied Behavior Analysis, 10(1), 103 to 116.
  2. McHugh, M. L. (2012). Interrater reliability: the kappa statistic. Biochemia Medica, 22(3), 276 to 282.
  3. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. Standards for Educational and Psychological Testing (2014), on standardized administration and scorer training.
  4. Protection of Human Subjects, 45 CFR Part 46 (training of research staff in human subjects protections, as required by institutions).

The example collector is made up. The record applies your protocol’s training steps and cutoff. Support: team@caveatacademic.com.