Estimate a population proportion or mean from a sample survey and develop a margin of error by simulation

Learning goal: Use data from a sample survey to estimate a population mean or proportion, and develop a margin of error for the estimate through the use of simulation models for random sampling.

The estimate. A hypothetical district has 6,000 high-school students and wants to know what proportion favor a later school start time. It selects 100 students at random from its enrollment list and asks each one; 62 say yes. The sample proportion p̂ = 62 ÷ 100 = 0.62 is the estimate of the population proportion p. Nobody believes p is exactly 0.62: a second random sample of 100 would give a slightly different count, perhaps 58 or 66. The question that a margin of error answers is: how far from the population proportion is a sample proportion like this one likely to be?

A simulation model for random sampling. We cannot draw more real samples, but we can imitate random sampling with a model. Suppose the population proportion really were 0.62, our best available guess. Draw a simulated sample of 100 by generating 100 random numbers between 0 and 1 and counting how many are below 0.62; that count over 100 is one simulated p̂. Repeat 1,000 times and look at how the simulated proportions spread out. The figures quoted are illustrative of one run: they were checked for plausibility against the exact distribution, not recorded from a reproducible run, and a real run gives slightly different figures. The 1,000 simulated proportions ranged from 0.47 to 0.80. Grouped by number of yes answers: 46–49 in 5 samples, 50–53 in 38, 54–57 in 115, 58–61 in 300, 62–65 in 306, 66–69 in 166, 70–73 in 60, 74–77 in 9, 78–81 in 1. The pile is centered at 0.62 and is roughly symmetric and bell-shaped, and the standard deviation of the 1,000 simulated proportions is 0.0487.

Turning spread into a margin of error. The middle 95% of the simulated proportions ran from 0.52 (the 2.5th percentile) to 0.72 (the 97.5th percentile), that is, within about 0.10 of the assumed 0.62. Equivalently, two standard deviations is 2 × 0.0487 = 0.097, about 0.10. So sample proportions from random samples of 100 land within about 10 percentage points of the population proportion 95% of the time. Turn that around: we estimate that the population proportion is within about 10 points of our observed 0.62. Two standard deviations works as a margin here because the simulated distribution is roughly symmetric and mound-shaped, so always look at it first: when a sample has very few yes or very few no answers the simulated distribution is lopsided, and a sample that was all yes would show no spread at all, suggesting a certainty that the data cannot give. The estimate is reported as 62% with a margin of error of 10 percentage points, or the interval 52% to 72%. The interpretation is about the method: intervals built this way from random samples of 100 capture the population proportion about 95% of the time. Any single interval either contains p or does not; we do not know which, which is why the word "confidence" rather than "probability" is attached to the interval. (A formula exists for this standard deviation, √(p(1 − p) ÷ n) = √(0.62 × 0.38 ÷ 100) = 0.0485, and it matches the simulation; the simulation is the reasoning behind it.)

Sample size. The margin of error shrinks as the sample grows, but slowly. Quadrupling the sample to 400 halves the standard deviation, to about 0.024, so the margin becomes about 5 points. Halving the margin costs four times the interviews. That is why surveys of a few hundred to a few thousand people are common: they give margins of a few points, and further precision costs too much.

Estimating a mean. The same reasoning works for a population mean, with a different simulation model. Thirty students were randomly selected from the same district and asked how many minutes of recreational screen time they had the previous day. The hypothetical responses are: 95, 120, 60, 180, 150, 75, 210, 90, 135, 45, 165, 100, 240, 110, 85, 130, 70, 155, 200, 115, 50, 140, 175, 105, 65, 190, 125, 80, 145, 160. The sum is 3,765 minutes, so x̄ = 125.5 minutes, and the sample standard deviation is s = 50.3 minutes. The estimate of the population mean is 125.5 minutes. To find how much sample means vary, we need a population to sample from, and the best stand-in we have is the sample itself: draw 30 values from these 30 at random with replacement (so a value can be chosen more than once), compute the mean, and repeat 1,000 times. This resampling model treats the sample as a miniature of the population. One run gave 1,000 resample means with a standard deviation of 9.09 minutes, so the margin of error is about 2 × 9.09 = 18.2, about 18 minutes: the population mean is estimated as 125.5 ± 18 minutes, roughly 107 to 144 minutes. (The formula s ÷ √n = 50.3 ÷ √30 = 9.19 agrees.)

What a margin of error does and does not cover. The margin of error describes random sampling variability only: the wobble that comes from which individuals happened to be selected. It says nothing about bias. If the 100 students had been chosen from one lunch period, or if 40 of the selected students had refused to answer, or if the question had been worded to push a yes, the sample proportion would be off in a direction the margin of error cannot see, and reporting "±10 points" would give false reassurance. A margin of error attached to a voluntary online poll is meaningless, because the simulation model of random sampling does not describe how that poll's respondents were chosen. The margin is also not a statement about the sample: 62% of the sample favored the change with no uncertainty at all. The uncertainty is about the 6,000.

Reporting. A complete report of a survey estimate states the population, how the sample was selected, the sample size, the response rate, the statistic, the margin of error and how it was obtained. For the district: "Of 100 students selected at random from the enrollment list, all of whom responded, 62% favored a later start; a simulation of random samples of this size gives a margin of error of about 10 percentage points, so the population percentage is estimated between 52% and 72%."

Worked proportion with margin: A random sample of 100 from the district gives 62 yes. Estimate: p̂ = 0.62. Simulation under p = 0.62: 1,000 samples of 100, standard deviation of the sample proportions 0.0487, middle 95% from 0.52 to 0.72. Margin of error: 2 × 0.0487 ≈ 0.10. Report: 62% ± 10 points, interval 52% to 72%. The statement "about 95% of random samples of 100 give a proportion within 10 points of the population proportion" is what justifies the interval.

Worked interpretation, right and wrong: Right: "we estimate that between 52% and 72% of the district's students favor a later start; the method captures the true percentage about 95% of the time." Wrong: "95% of students are in the interval" (the interval is about a proportion, not about students) and "there is a 95% chance p is between 52% and 72%" (p is a fixed number; the chance belongs to the method, before the sample is drawn). Also wrong: "62% of the sample, give or take 10%" (the sample percentage is known exactly).

Worked mean by resampling: The 30 screen-time values have x̄ = 125.5 and s = 50.3 minutes. One resample: pick 30 values at random with replacement from the 30, say 95, 95, 60, 240, ... and average them; a resample mean might be 131.2. After 1,000 resamples the means have standard deviation 9.09 minutes. Margin of error 2 × 9.09 ≈ 18 minutes. Estimate: the district's mean recreational screen time is about 125.5 minutes, between about 107 and 144 minutes. The interval is wide because the sample is small and the data are spread out (s = 50.3).

Worked sample-size decision: The district wants a margin of 5 points instead of 10 for the start-time question. Halving the margin requires halving the standard deviation of sample proportions, which requires four times the sample: 400 students. Check: √(0.62 × 0.38 ÷ 400) = 0.024, and 2 × 0.024 ≈ 0.05. A margin of 2.5 points would need about 1,600 students; at some point the cost outweighs the precision. These calculations treat the sample as a small share of the population, as 100 or 400 of 6,000 are. A sample of 1,600 is over a quarter of the 6,000 students, and sampling so large a share without replacement makes estimates somewhat more precise than the rule says, so a simulation that draws without replacement from the actual list would show that slightly fewer students suffice.

Represent the simulation: Using a spreadsheet, build one column of 100 random numbers and a cell that counts how many are below 0.62. Recalculate 50 times, recording the count each time, and make a dot plot of your 50 sample proportions. Mark 0.62, 0.52 and 0.72 on the axis and count how many of your 50 fall inside; compare with the 1,000-run figures (standard deviation 0.0487; middle 95% from 0.52 to 0.72). Write one sentence explaining why your dot plot and the bar chart have the same shape and center.

Independent performance: Run a real sample survey of at least 40 randomly selected people from a defined list (a school directory or a list of student numbers, large enough that 40 is a small share of it), asking one yes/no question and one numerical question. (1) Describe the random selection procedure and the response rate. (2) Report p̂ and x̄. (3) Simulate random sampling for the proportion (at least 200 simulated samples at your observed p̂) and resampling for the mean (at least 200 resamples with replacement), and report the standard deviation of each simulated distribution. (4) Look at each simulated distribution; if it is roughly symmetric and mound-shaped, state each estimate with its margin of error as two standard deviations and as an interval, if it is lopsided, say so and report the middle 95% of the simulated values instead; and if it shows no spread at all (for example, every simulated proportion is 1 because every person sampled said yes), say that this simulation cannot produce a useful interval, because a zero-width interval would claim a certainty the data cannot give, and report the count and the sample size without a margin. (5) Write a paragraph on what the margins do not cover: non-response, wording, and anyone your list omitted. Success means the population and list are named, the two simulation models are described so a reader could repeat them, each margin is derived from a simulated standard deviation, and the interpretation is about the population rather than the sample.