Vocations · Job Skills

Did the training
change the work?

A good day of training is not the same as a change at work. Tag each follow-up item with the level it measures, add test scores and one business measure from before and after, and enter the averages once responses are in. The tool shows the gain, the average at each level and the gap between how much people liked it and how much they use it.

Add a training programWhat you get
$12one-time · every device you sign in on
Buy · $12

Included with Caveat Classroom and Whole-school licenses. Or get it in the Job search kit, $29 for 45 items.

Training programs

The worked example is made up: a forklift refresher with good transfer, a customer service course people liked but rarely use and a software rollout with no results measure yet. Saved on this device only.

Loading the program form. If nothing appears, turn on JavaScript.

What you get

Vocations · Job Skills$12 · included with Classroom and Whole-school licenses

What you finish with
A record per training program: completions and follow-up responses, test scores before and after with the normalized gain, a business measure before and after, each survey item’s average tagged by level, five design checks, and flags for a liked-but-not-used gap, missing levels, low response and small gains. Prints the follow-up survey.
Who it is for
Training managers, HR and workforce development staff, career and technical education partners, and supervisors who run on-the-job training.
Format
Browser tool with a saved record per program. Prints the follow-up survey, one program’s results, every program in a table or a blank form; downloads a spreadsheet (.csv); exports and imports a file.
Access
$12 one-time, yours on every device you sign in on; the tool opens once you are signed in with it. Included with Caveat Classroom and Whole-school licenses. Also in the Job search kit ($29) and the Vocations Suite pass ($99), and 1 more bundle. Everything you type stays in this browser; nothing is sent to Caveat.
Where it lives
The Job Skills space in Caveat Vocations.
Support
Questions about access, an order or a correction: team@caveatacademic.com.

How to use it

  1. Write items for all four levels.
  2. Record test scores and a baseline.
  3. Send the survey 30 to 90 days later.
  4. Enter the averages.
  5. Read the transfer gap.

Add a training program

Measuring past the last day

Four levels of evidence1 ReactionDid people find ituseful and well run?Asked on the lastday.2 LearningCan they do orexplain it now?A test before andafter.3 BehaviorDo they use it onthe job?Asked 30 to 90 dayslater, of them andtheir supervisors.4 ResultsDid the work measuremove?Errors, incidents,time, sales, with abaseline.Four levels of evidence1 ReactionDid people find it useful and well run?Asked on the last day.2 LearningCan they do or explain it now?A test before and after.3 BehaviorDo they use it on the job?Asked 30 to 90 days later, of them and theirsupervisors.4 ResultsDid the work measure move?Errors, incidents, time, sales, with abaseline.
Most training surveys stop at level 1. Levels 3 and 4 are what a manager or a funder will ask about.

Normalized gain

A rise from 40% to 85% and a rise from 85% to 95% are both good, but they are not the same size of change. The normalized gain divides the points gained by the points that were possible to gain: (after minus before) divided by (100 minus before). Around 0.3 or less is a small gain; 0.7 or more is a large one. It was introduced for physics teaching and works for any before-and-after test on the same scale.

When people liked it but do not use it

Research on training transfer finds that what happens back at work matters as much as the course: a chance to use the skill soon, the tools to do it and a supervisor who expects it. A reaction average well above the behavior average is a sign to look there first, before rewriting the course. A business measure can move for other reasons too, so note what else changed.

Sources

  1. Kirkpatrick, J. D., and Kirkpatrick, W. K. (2016). Kirkpatrick’s Four Levels of Training Evaluation. ATD Press.
  2. Hake, R. R. (1998). Interactive-engagement versus traditional methods: a six-thousand-student survey of mechanics test data. American Journal of Physics, 66(1), 64 to 74.
  3. Saks, A. M., and Belcourt, M. (2006). An investigation of training activities and transfer of training in organizations. Human Resource Management, 45(4), 629 to 648.
  4. Grossman, R., and Salas, E. (2011). The transfer of training: what really matters. International Journal of Training and Development, 15(2), 103 to 120.

The worked example is made up, including every program and figure. The 30% response and 0.3 gain thresholds and the 0.5-point gap are working rules stated on the page, not standards. Nothing you type is sent to Caveat. Support: team@caveatacademic.com.