Teaching · For Teachers

Find the confusing items while they are still easy to fix.

Choose your pilot methods, run them with a few people like your real audience, and log every item that caused trouble. The summary ranks the problem items, shows whether the survey runs long, and keeps a list of fixes still to make.

Start a pilot planWhat you get
$12one-time · every device you sign in on
Buy · $12

Included with Caveat Classroom and Whole-school licenses. Or get it in the For Teachers toolkit, $79 for 72 items.

The pilot

One record per survey and pilot round. Pilot respondents are counted, never named. The example is made up.

Loading the form. If nothing appears, turn on JavaScript.

What you get

Teaching · For Teachers$12 · included with Classroom and Whole-school licenses

What you finish with
For each pilot round, the methods and respondents, timing against the target, a ranked log of problem items by kind with fixes, and a list of what to finish before launch.
Who it is for
Teachers and school teams who write surveys for students, families or staff, graduate students and program evaluators.
Format
Browser form. Several saved pilot rounds per survey; printed with the summary or as a blank trouble log for interviews; CSV of the trouble log; export and import a file.
Access
$12 one-time, yours on every device you sign in on; the tool opens once you are signed in with it. Included with Caveat Classroom and Whole-school licenses. Also in the For Teachers toolkit ($79) and the Teaching Suite pass ($199), and 1 more bundle. Everything you type stays in this browser; nothing is sent to Caveat.
Where it lives
The For Teachers space in Caveat Teaching.
Support
Questions about an order, access or a correction: team@caveatacademic.com.
?find it before they do
Drawn for this page. A pilot finds the confusing item while it can still be fixed, before hundreds of answers depend on it.

How to use it

  1. Start a record with a survey code and the pilot date.
  2. Choose the methods and note who took part and how many.
  3. Time each pilot respondent and enter the median and slowest times.
  4. Log every item that caused trouble, the kind of problem, how many hit it and the fix.
  5. Revise, then run a second short round if many items changed.

Start a pilot plan

What the pilot is for

Three kinds of pilot

An expert review catches items that miss what you mean to measure. Cognitive interviews, where people think aloud or answer follow-up probes about each item, show how real readers understand the words. A small field test checks timing, skipping and the flow of the whole survey.

Reading the results

An item that trips up several of a dozen pilot respondents will trip up many more in the real survey. Double-barreled items and missing answer options are common and easy to fix. A survey that runs long loses answers at the end; cut or simplify before launch.

What this planner is not

It does not test reliability or validity statistically. It records a qualitative pilot so the survey that goes out is clearer.

Sources

  1. Willis, G. B. Cognitive Interviewing: A Tool for Improving Questionnaire Design (2005).
  2. Dillman, D. A., Smyth, J. D., and Christian, L. M. Internet, Phone, Mail, and Mixed-Mode Surveys: The Tailored Design Method, 4th edition (2014), on pretesting.
  3. Fowler, F. J., Jr. Survey Research Methods, 5th edition (2014), on question design and pretesting.
  4. Gehlbach, H., and Brinkworth, M. E. (2011). Measure twice, cut down error: a process for enhancing the validity of survey scales. Review of General Psychology, 15(4), 380 to 387.

The example survey is made up. The planner records a pilot; it does not compute reliability or validity. Support: team@caveatacademic.com.