Comparing Two Treatments
Aligned to S-IC.B.5 — Common Core State Standards for Mathematics.
What this lesson teaches
In a randomized experiment, compare the mean outcomes of two treatment groups. A difference is likely significant if it is larger than the variation you would expect from chance alone, which a simulation or randomization test can check.
Worked examples
Example 1
Group A (new drug) averages 8 fewer sick days than Group B (placebo). If randomly reshuffling the group labels rarely produces a gap as large as 8, then the observed difference is statistically significant.
Example 2
Two teaching methods are compared on 30 students, randomly split 15 and 15. Method A averages 82, Method B averages 76, a gap of 6 points. Pool all 30 scores and re-split them at random 1,000 times. If 180 of those random splits produce a gap of 6 or more, the p-value is 0.18 — common enough by chance that the methods cannot be called different.
Example 3
Common mistake: reading a large p-value as proof that the treatments are identical. A p-value of 0.18 means the data is consistent with pure chance, not that the difference is zero. A small study often cannot detect a real effect: the same 6-point gap measured on 300 students instead of 30 might give a p-value near 0.01. Failing to find a difference and finding no difference are different conclusions.
Practice questions
- What does it mean for a difference to be statistically significant?
- Why compare the means of the treatment groups rather than individuals?
- How can a simulation test whether a difference is due to chance?
Watch the lesson
Every lesson comes with a video taught in English and Spanish — the same video the QR code in the printed workbook opens.
▶ Watch this lessonEn español
Comparar dos tratamientos
En un experimento aleatorizado, compara los resultados promedio de dos grupos de tratamiento. Una diferencia probablemente es significativa si es mayor que la variación que esperarías solo por azar, lo cual una simulación o prueba de aleatorización puede comprobar.
Ejemplo 1: El grupo A (medicamento nuevo) promedia 8 días de enfermedad menos que el grupo B (placebo). Si reordenar al azar las etiquetas de los grupos rara vez produce una diferencia tan grande como 8, entonces la diferencia observada es estadísticamente significativa.
Ejemplo 2: Se comparan dos métodos de enseñanza en 30 estudiantes, divididos al azar en 15 y 15. El método A promedia 82, el B promedia 76, una diferencia de 6 puntos. Junta los 30 resultados y vuelve a dividirlos al azar 1,000 veces. Si 180 de esas divisiones producen una diferencia de 6 o más, el valor p es 0.18 — bastante común por azar como para no poder llamar distintos a los métodos.
Ejemplo 3: Error común: leer un valor p grande como prueba de que los tratamientos son idénticos. Un valor p de 0.18 significa que los datos son compatibles con el puro azar, no que la diferencia sea cero. Un estudio pequeño a menudo no puede detectar un efecto real: la misma diferencia de 6 puntos medida en 300 estudiantes en vez de 30 podría dar un valor p cercano a 0.01. No hallar una diferencia y hallar que no hay diferencia son conclusiones distintas.
More Statistics & Probability lessons
Teach this lesson today
An AI tutor that guides with questions instead of just giving answers — all K–12 core subjects, every child in the family.
Start free — no card required