Distinguish sample surveys, experiments and observational studies, and explain the role of randomization in each

Learning goal: Recognize the purposes of and differences among sample surveys, experiments and observational studies, and explain how randomization relates to each: random selection for generalizing to a population, random assignment for concluding that a treatment causes an effect.

Three ways to collect data. Statistical studies differ in what the investigator controls. In a sample survey, individuals are selected from a population and a characteristic is recorded for each, usually by asking questions or taking a measurement, without changing anything about them. Its purpose is to estimate a population parameter, such as the proportion of a city's voters who support a levy or the mean number of hours its students sleep. In an experiment, the investigator deliberately imposes a treatment on the individuals, called experimental units or, when human, subjects, and records a response. Its purpose is to compare treatments and decide whether a treatment causes a change in the response. In an observational study, the investigator records both an explanatory variable and a response for each individual but does not assign the explanatory variable; the individuals arrived with their values of it already set. Its purpose is to describe an association, and it is used when assigning the explanatory variable is impossible or unethical.

Two different uses of randomization. The word randomization covers two separate procedures that answer two separate questions. Random selection means choosing which individuals from the population enter the study by a chance mechanism, such as a random-number generator applied to a list of the whole population. Random selection lets the results be generalized to the population, because a randomly selected sample has no systematic tilt and its statistics vary in a predictable way. Random assignment means deciding by a chance mechanism which treatment each unit in the study receives, for example by shuffling numbered cards. Random assignment lets a difference in the response be attributed to the treatment, because it spreads every other influence, known or unknown, evenly between the groups on average. A lurking variable such as motivation or soil quality cannot line up systematically with the treatment when a coin decided who got which treatment; any imbalance left is due to chance, which is what the later simulations measure.

How each study type uses randomization. A sample survey relies on random selection: its whole value is that a few hundred randomly chosen people can stand in for a population of many thousands. It imposes no treatment, so random assignment does not arise, and it cannot establish cause. An experiment relies on random assignment to make causal conclusions; if its subjects were also randomly selected from a population, the causal conclusion can be generalized to that population, but many experiments use volunteers, so the conclusion applies to units like the ones studied and the generalization is an argument, not a statistical guarantee. An observational study, by definition, does not randomly assign the explanatory variable. It may use random selection, in which case its association generalizes to the population, but the association could be explained by a lurking variable, so it cannot by itself show cause. The two questions, "can I generalize?" and "can I claim cause?", are answered by the two randomizations separately.

Hypothetical survey. A city with 40,000 registered voters wants to know what proportion favor a park levy. A random sample of 300 voters is drawn from the registration list and interviewed; 174 favor it, so p̂ = 0.58. The random selection is what allows 0.58 to stand for the city. Note what could still go wrong: if only 200 of the 300 answer and the 100 who refuse differ from the rest, the sample is no longer a random picture of the population, whatever the original selection. Random selection must be preserved by a high response rate and neutral question wording.

Hypothetical experiment. A teacher wants to know whether a ten-minute walk before a quiz raises scores. Forty student volunteers are numbered 1 to 40; a random-number generator picks 20 numbers, and those students walk while the other 20 sit quietly; everyone then takes the same quiz. Random assignment ensures that the students who are fit, anxious, well-rested or strong at the subject are spread between the two groups by chance, so if the walkers' mean score is clearly higher, the walk is the plausible cause. Good experiments add a control group (the sitters, so that the walk is compared with something), keep everything else the same, and, where possible, keep the person scoring the quiz unaware of who walked. Because the 40 were volunteers rather than randomly selected, the causal conclusion applies to students like these; whether it holds for all students is a separate argument.

Hypothetical observational study. A researcher compares the mathematics grades of students who chose to take private music lessons with those who did not, and finds the music students score higher on average. Nothing was assigned: families chose lessons. Families who pay for lessons may also have more time, money or emphasis on schoolwork, and any of those could raise both lesson-taking and grades. The study describes an association; it cannot show that music lessons cause higher grades. Turning it into an experiment would mean assigning lessons at random, which is expensive but possible; in cases like the effect of smoking on health, assignment is unethical, so observational studies with careful adjustment for known lurking variables are the best available, and their causal claims stay more tentative.

Sorting a study. Two questions, not one. First: did the investigator impose a treatment? If yes, it is an experiment; ask whether treatments were randomly assigned. If no, it is observational. Second: how were the individuals chosen? A sample survey is a way of collecting data, a sample selected from a population and asked or measured; ask whether selection was random. The categories overlap: a survey that records an explanatory and a response variable and examines their association is also an observational study, because no treatment is imposed. In this lesson a study is called a sample survey when its purpose is to estimate a population characteristic, and an observational study when its purpose is to examine an association between an explanatory variable and a response; for either, name the lurking variables that could explain any association.

Worked classification 1: A hypothetical hospital records, for 2,000 randomly selected patient files, whether the patient drank coffee daily and whether they reported sleep problems. No treatment was imposed and coffee-drinking was the patient's own habit, so this is an observational study. Random selection of files lets the association generalize to the hospital's patients; the absence of random assignment means sleep problems could be linked to coffee through stress, shift work or age rather than caused by it.

Worked classification 2: A seed company numbers 60 pots, uses a random-number generator to choose 30 pots for a new fertilizer and gives the other 30 the standard fertilizer, then measures plant height after six weeks. A treatment is imposed and randomly assigned: a randomized experiment. If the new-fertilizer mean is clearly higher, the fertilizer is the plausible cause. The 60 pots were not a random sample of all growing conditions, so extending the result to a farm's fields is an argument about similarity, not a statistical inference.

Worked classification 3: A school board emails all 3,000 parents a link and 420 reply, 70% opposing a schedule change. This is a sample survey, but the sample is a voluntary response, not a random selection: parents with strong feelings are more likely to reply. The 70% cannot be generalized to all parents. Redesign: select 300 parents at random from the full list, contact each individually and follow up with non-responders.

Worked redesign from observation to experiment: Observation: students who use a study app report higher quiz scores. Experiment: from 50 volunteers, randomly assign 25 to use the app for two weeks and 25 to study the same material without it, then give one quiz. Random assignment balances prior ability and motivation across the groups; comparing the two means then tests the app itself. The app cannot be assigned to the whole school population, so the conclusion is about students like the volunteers.

Represent the designs: Make a three-column chart headed sample survey, experiment and observational study. Under each, write its purpose, what the investigator controls, which randomization it uses, what conclusion it can support and one hypothetical example from this lesson. Then draw a two-by-two grid with rows "randomly selected? yes/no" and columns "randomly assigned? yes/no" and place each of the four worked examples in its cell with a note on what conclusion is allowed there.

Independent performance: Write a research question about something in your school or community. Design all three studies for it: a sample survey with a genuine random-selection procedure from a named list, a randomized experiment with a control group and a stated assignment mechanism, and an observational study naming two lurking variables. For each, state what conclusion the design supports and what it cannot support. Then choose the design you would actually run and defend the choice in a paragraph considering cost, ethics and what you need to conclude. Success means each randomization is described as a procedure someone else could carry out, the generalization and causation questions are answered separately for each design, and no design claims more than its randomization allows.