Compare two treatments in a randomized experiment and use simulation to decide whether the difference is statistically significant

Learning goal: Use data from a randomized experiment to compare two treatments, and use a simulation of the random assignment to decide whether the difference between the sample statistics is statistically significant.

The experiment. A hypothetical gardening class tests whether a compost-enriched potting mix makes tomato seedlings grow taller than the standard mix. Twenty seedlings of the same variety and age are numbered 1 to 20; a random-number generator selects 10 numbers, and those seedlings are planted in the compost mix (treatment A) while the other 10 go into the standard mix (treatment B). Everything else is kept the same: same bench, same light, same watering. After four weeks the growth in centimeters is measured. The invented data are:

Treatment A (compost mix): 18.2, 21.5, 19.8, 23.1, 20.4, 22.7, 17.9, 24.0, 21.1, 19.6. Sum 208.3 cm, mean 20.83 cm.

Treatment B (standard mix): 16.4, 19.9, 18.1, 17.5, 20.8, 15.9, 18.7, 19.3, 17.0, 18.4. Sum 182.0 cm, mean 18.20 cm.

The difference between the sample means is 20.83 − 18.20 = 2.63 cm in favor of the compost mix. The two groups also overlap: the tallest standard-mix plant (20.8) beat five of the compost plants (17.9, 18.2, 19.6, 19.8 and 20.4). So the question is not whether A's mean is higher, which it is, but whether a difference of 2.63 cm is more than the random assignment alone could easily produce.

The question behind significance. Suppose the mix made no difference at all. Then each seedling would have grown by exactly the same amount whichever group it had been assigned to; the 20 growth values were fixed by the seedlings, and the only thing chance did was decide which 10 of those values got the label A. Under that "no effect" assumption, the observed difference of 2.63 cm is just the result of one particular random split of 20 fixed numbers. Some random splits would put more of the naturally vigorous seedlings in group A and produce a positive difference; others would do the reverse. If differences as large as 2.63 cm arise often from random splitting alone, the experiment has found nothing the assignment could not explain. If they arise rarely, the difference is statistically significant: if the mix had no effect, random assignment alone would rarely produce a gap this large. That is a probability computed by assuming no effect; it is not the probability that chance caused the observed gap. Since random assignment balanced everything else on average, the mix is then the plausible cause.

The re-randomization simulation. The simulation imitates exactly what the "no effect" assumption says happened. Write the 20 growth values on cards, shuffle, deal 10 to a pile labeled A and 10 to a pile labeled B, compute mean A minus mean B, record it, and repeat. A computer does this thousands of times in a second; the figures below are illustrative of one run of 1,000 shuffles: they were checked for plausibility against the exact shuffle distribution, not recorded from a reproducible run, and a real run gives slightly different figures. The 1,000 simulated differences ranged from −3.03 to 2.71 cm with standard deviation 0.99 cm, and were centered on 0, as they must be when the labels are meaningless. Grouped in 1-cm bins, from −4 to −3 there was 1 difference, from −3 to −2 there were 19, from −2 to −1 there were 150, from −1 to 0 there were 334, from 0 to 1 there were 325, from 1 to 2 there were 149, from 2 to 3 there were 22, and from 3 to 4 there were none.

The decision. Count the simulated differences at least as extreme as the observed one in either direction, that is, with absolute value at least 2.63. The run produced 4 of them (1 at or above +2.63 and 3 at or below −2.63), a proportion of 4 ÷ 1,000 = 0.004. Random assignment alone produces a gap this large about four times in a thousand. That is rare by any usual standard, so the difference is statistically significant, and the conclusion is that the compost mix caused greater growth in these seedlings. The chain of reasoning has three links, and each is needed: the treatments were randomly assigned, so lurking variables were balanced on average and could differ between groups only by chance; the simulation shows random assignment rarely makes a gap this size; therefore the treatment is the plausible cause of the gap.

A difference that is not significant. A second hypothetical experiment tests two memorization methods with 16 volunteers randomly assigned, 8 to each. Method 1 scores: 12, 15, 9, 14, 11, 13, 10, 16, mean 12.5. Method 2 scores: 11, 14, 10, 12, 15, 9, 13, 12, mean 12.0. Difference 0.5 points. Re-randomizing the 16 scores 1,000 times, one run gave a difference of at least 0.5 in absolute value in 746 of the 1,000 shuffles, a proportion of 0.746. Random assignment produces gaps this size three times out of four; the observed difference is exactly the kind chance makes. The difference is not statistically significant, and the correct conclusion is that the data are consistent with the methods having the same effect. That is not the same as showing they have the same effect: a small real difference could be hidden by the variability of 16 scores. "Not significant" means the experiment could not distinguish the treatments, not that they are identical.

Conventions and cautions. A proportion below 0.05 is the usual threshold for calling a result significant, but it is a convention; 0.004 is clearly rare, 0.746 clearly common, and a result near 0.05 calls for a larger experiment rather than a verdict. Significance is about whether an effect is detectable, not whether it matters: a 2.63 cm gain in four weeks may or may not be worth the cost of compost, and that is a gardening judgment, not a statistical one. The extreme region should be chosen before the count: "at least as extreme in either direction" (two-sided) is the default when either treatment could have been better. Finally, the seedlings were not a random sample of all tomato seedlings, so the causal conclusion applies to seedlings like these under these conditions; extending it to a farm is a separate argument.

Worked observed statistic: Treatment A sums to 18.2 + 21.5 + 19.8 + 23.1 + 20.4 + 22.7 + 17.9 + 24.0 + 21.1 + 19.6 = 208.3, mean 20.83 cm. Treatment B sums to 16.4 + 19.9 + 18.1 + 17.5 + 20.8 + 15.9 + 18.7 + 19.3 + 17.0 + 18.4 = 182.0, mean 18.20 cm. Observed difference A − B = 2.63 cm. The statistic to simulate is this difference of means, and the extreme region is any shuffle giving a difference of at least 2.63 in absolute value.

Worked one shuffle by hand: Shuffle the 20 values and deal 10 to A: suppose they are 16.4, 21.5, 18.1, 23.1, 17.5, 15.9, 21.1, 18.7, 19.6, 20.8, which sum to 192.7, mean 19.27; the remaining ten (18.2, 19.8, 20.4, 22.7, 17.9, 24.0, 19.9, 19.3, 17.0, 18.4) sum to 197.6, mean 19.76. This shuffle's difference is 19.27 − 19.76 = −0.49 cm, a typical value near 0. The observed 2.63 would need most of the large values to land in A by chance, which the 1,000-shuffle run shows happens rarely.

Worked decision for the seedlings: Simulated differences with |difference| ≥ 2.63: 4 of 1,000, proportion 0.004. Since 0.004 is far below the 0.05 convention, the observed difference is statistically significant. Because the mix was randomly assigned, the compost mix is concluded to have caused the extra growth in these seedlings. Reported: "the compost mix produced a mean gain of 2.63 cm over the standard mix; a re-randomization simulation gave a difference this large in 0.4% of shuffles."

Worked decision for the memory methods: Means 12.5 and 12.0, difference 0.5. Simulated |difference| ≥ 0.5 in 746 of 1,000 shuffles, proportion 0.746. The observed gap is ordinary under random assignment, so it is not statistically significant. Conclusion: no evidence that the methods differ; the experiment cannot rule out a small real difference, so the honest wording is "the data are consistent with no difference," not "the methods are equally effective."

Represent the simulation: Write the 20 seedling values on cards. Shuffle and deal 10 to A ten times, computing the difference of means each time, and plot your ten differences on a number line from −4 to 4. Mark the observed 2.63 and −2.63. Beside your plot copy the 1,000-shuffle bin counts (1, 19, 150, 334, 325, 149, 22, 0) and explain why none of your ten shuffles is likely to reach ±2.63 when only 4 of 1,000 did.

Independent performance: Design and run a small randomized experiment with two treatments and at least 8 units per group, for example two paper-airplane folds and flight distance, or two rest lengths and the number of arithmetic problems solved in a minute. (1) Describe the random assignment mechanism and what was held constant. (2) Report both groups' data, both means and the observed difference. (3) Re-randomize at least 200 times with cards or a spreadsheet, keeping every shuffle's difference. (4) Report the proportion of shuffles at least as extreme as the observed difference in either direction and state whether the difference is statistically significant. (5) Write the conclusion, including whether causation is justified, whether the result generalizes beyond your units and whether the size of the effect matters in practice. Success means the extreme region is defined before counting, the proportion comes from your recorded shuffles, and "not significant" is not written as "no effect".