Teaching · For Teachers · Research and data

Is the gap between
two groups bigger
than chance?

Paste each group’s scores. The calculator runs the t-test researchers use, shows every step, and pairs the p-value with the size of the difference and its confidence interval, so a small study is read for what it can and cannot show.

Run the testWhat you get
$12one-time · every device you sign in on
Buy · $12

Included with Caveat Classroom and Whole-school licenses. Or get it in the For Teachers toolkit, $79 for 72 items.

The t-test calculator

The worked example is made up: two sections taught the same unit, one with weekly retrieval quizzes. Use codes, not names. Nothing you type leaves this browser.

Loading the calculator. If nothing appears, turn on JavaScript.

What you get

Teaching · For Teachers$12 · included with Classroom and Whole-school licenses

What you finish with
t, degrees of freedom and the two-tailed p-value, the difference in means with its confidence interval, Cohen’s d and Hedges’ g, group summaries and the steps. It prints as a summary and downloads as a spreadsheet.
Who it is for
Teachers doing action research, data teams and instructional leaders, school psychologists and graduate students in education.
Format
Browser calculator for a phone, tablet or computer, by touch or keyboard. Up to 2,000 scores a group. Prints a summary; downloads a spreadsheet (.csv); export and import a file.
Access
$12 one-time, yours on every device you sign in on; the tool opens once you are signed in with it. Included with Caveat Classroom and Whole-school licenses. Also in the For Teachers toolkit ($79) and the Teaching Suite pass ($199), and 1 more bundle. Everything you type stays in this browser; nothing is sent to Caveat.
Where it lives
The For Teachers space in Caveat Teaching.
Support
Questions, a correction or a request: team@caveatacademic.com.

How to use it

  1. Name the measure and the two groups.
  2. Paste each group’s scores.
  3. Keep Welch unless you have a reason to assume equal spread; choose a confidence level.
  4. Read the p-value together with the difference, its interval and d.
  5. Print the summary or download the spreadsheet.

Run the test

How the test works

A difference measured against its own noise.

The worked example, from means to a p-valueDifference in means6.3784.92 − 78.55Standard error2.449√(s²A/12 + s²B/11)t2.606.37 ÷ standard errorTwo-tailed p= .017df 21.0, WelchThe worked example, from means to a p-valueDifference in means6.3784.92 − 78.55Standard error2.449√(s²A/12 + s²B/11)t2.606.37 ÷ standard errorTwo-tailed p= .017df 21.0, Welch
The retrieval-quiz section averaged 6.37 points more; set against the noise expected from groups this size and spread, that gives t = 2.60 and a two-tailed p = .017.

Welch or Student

Both divide the difference in means by its standard error and read the result against a t distribution. Student’s test pools the two variances and assumes they are equal; Welch’s does not, and adjusts the degrees of freedom instead. Welch’s test loses almost nothing when the variances are equal and protects you when they are not, so it is the default here.

Reading the result

The p-value is the chance of a difference at least this large if the groups truly did not differ. It is not the chance the finding is true, and it shrinks with larger samples even when the difference stays small. Report the difference with its confidence interval and an effect size. Two classes are not random samples, so a difference may come from the classes as much as from what was done in them.

Sources

  1. Welch, B. L. (1947). The generalization of “Student’s” problem when several different population variances are involved. Biometrika, 34(1–2), 28–35.
  2. NIST/SEMATECH e-Handbook of Statistical Methods, section 1.3.5.3, Two-Sample t-Test for Equal Means (itl.nist.gov).
  3. Institute of Education Sciences, What Works Clearinghouse. Procedures and Standards Handbook, Version 5.0 (2022): Hedges’ g and the small-sample correction.
  4. Wasserstein, R. L., and Lazar, N. A. (2016). The ASA statement on p-values: context, process, and purpose. The American Statistician, 70(2), 129–133.

The worked example is made up. Use student codes, not names. Your scores stay in this browser. Support: team@caveatacademic.com.