Evaluate reports based on data

Learning goal: Evaluate reports based on data by asking who was studied and how they were chosen, what kind of study was done and what conclusion it can support, what was measured and how, whether uncertainty is stated, whether the numbers and graphs are presented honestly, and whether the conclusion stays within the data.

Why reports need evaluating. A report based on data arrives as a finished sentence: "78% oppose the bike lane", "tutoring raises grades by 0.4 points", "the new drink doubles alertness". The sentence is the end of a chain: a population was defined, individuals were chosen, something was measured, a statistic was computed, and someone wrote a claim. Each link can be weak, and the sentence hides the chain. Evaluating a report means rebuilding the chain and testing each link with the ideas of the earlier lessons. The questions below are the checklist; the three hypothetical reports that follow show how it is applied.

1. Who was studied and how were they selected? Identify the population the claim is about and the sample the data came from. Was the sample randomly selected from that population? A voluntary response sample (people who chose to click, phone in or reply), a convenience sample (whoever was nearby) or a sample from a different population (one city, one age group) cannot be generalized to the claimed population, whatever its size. How many were in the sample, and what proportion of those selected actually responded? A large sample does not repair a biased selection, and a low response rate can undo a good one.

2. What type of study was it, and does the conclusion match it? If the report claims that something causes something else, was there random assignment of the treatment? An observational study can show association only, and the evaluator should be able to name a lurking variable that could explain the association. A sample survey estimates a parameter and supports no causal claim at all. Reports routinely convert "associated with" into "leads to" or "boosts"; the evaluator converts it back.

3. What exactly was measured, and how? Read the question wording if it is given; "Do you support wasting money on a bike lane?" measures something different from "Do you support the proposed bike lane?" Check whether the measurement is the thing the headline names: "alertness" measured by a self-rating after ten minutes is not the same as performance at work. Check that both groups were measured the same way and by someone who did not know which group was which.

4. Is uncertainty reported? A sample estimate should come with a margin of error and the sample size behind it. A comparison should say whether the difference is statistically significant and how that was judged. The margin that matters for a comparison is the margin of the difference. For two independent samples it is larger than the margin of either estimate (about √2 times as large when the margins are equal); repeated measurements on the same people are not independent and need a different calculation. If the headline difference is smaller than that margin, the report has not shown that a real difference exists. If a difference is called significant, ask whether it is also large enough to matter.

5. Are the numbers presented honestly? Percentages need a base: "doubles the risk" may mean from 1 in 10,000 to 2 in 10,000, an absolute change of 0.01 percentage points. A percentage of a small group ("3 of 4 dentists") is not a percentage of all dentists. Averages can hide skew; a mean pulled up by one large value can misrepresent a typical case. Graphs mislead when the vertical axis starts far above zero so that a small difference looks enormous, when intervals on an axis are unequal, when a time window is chosen to start at a low point, or when area or pictures scale in two dimensions for a one-dimensional quantity.

6. Who did the study, and is it alone? A study conducted or funded by a party that benefits from its conclusion deserves closer reading, not automatic dismissal; the question is whether its methods survive the first five checks. One study is one piece of evidence; the evaluator asks whether other studies with different samples and designs agree.

7. Does the conclusion stay inside the data? Extrapolation beyond the range studied, generalization to groups not sampled, and claims about long-term effects from a short study are the usual ways a conclusion outruns its data.

Hypothetical report A. "78% of residents oppose the new bike lane, according to a poll of 1,240 people on a local news website." The population claimed is residents; the sample is website visitors who chose to vote, so this is voluntary response. People with strong feelings either way are the most likely to vote in such polls, and the direction of the bias is unknown. The 1,240 is large but irrelevant: no random selection means no basis for generalizing, and no margin of error can be meaningfully attached. Evaluation: the report tells us about the site's voting visitors, not residents.

Hypothetical report B. "Students who use the tutoring center earn grades 0.4 points higher on average; the center boosts achievement." This is an observational study: students chose to attend. Students who seek tutoring may be more organized or motivated, or may have parents who insist on it, and those traits could raise grades independently of the tutoring; alternatively, weaker students may attend and the 0.4 could understate a real benefit. Without random assignment the causal verb "boosts" is unsupported. The report also gives no sample size, no margin of error and no indication of whether 0.4 is significant. Evaluation: an association worth investigating with a randomized design; the causal claim is not established.

Hypothetical report C. "New energy drink doubles alertness, company study finds." Twelve employees of the company drank the product and rated their alertness from 1 to 10 before and after; the mean rose from 3 to 6. There is no control group, so the rise could reflect the passage of time, the expectation of an effect, or wanting to please the employer; there was no random assignment and the subjects were company employees, not a random sample of anyone. "Doubles" is a ratio of self-ratings on an arbitrary scale, and the report's bar chart starts its vertical axis at 2.5, making the 3-to-6 change fill the frame. Evaluation: the claim has no support the checklist recognizes.

A report that passes. This report is hypothetical. "In a random sample of 800 registered voters drawn from the county list, with 610 responding (76%), 56% favored the levy; the margin of error is 4 percentage points (two standard deviations, 2 × √(0.56 × 0.44 ÷ 610) ≈ 0.040, computed from the 610 who responded); question wording is published in the appendix." Population and list named, random selection, sample size and response rate given, statistic with margin, wording available. The evaluator can still note the 24% non-response as the main remaining risk, and that 56% ± 4 gives 52% to 60%, so if the non-responders do not differ much from the responders, the levy's lead appears real but modest.

Worked evaluation of a survey report: "A random sample of 400 students shows 52% prefer a longer lunch, up from 48% last year, so support is growing." Check the margin of error: a simulation of random samples of 400 at 0.5 gives a standard deviation of about 0.025, so the margin is about 5 points, and each year's estimate has one. The comparison needs the uncertainty of the difference, not of each estimate. For two independent random samples of 400, the standard deviation of the difference is √(0.25 ÷ 400 + 0.25 ÷ 400) ≈ 0.035, so its margin is about 2 × 0.035 ≈ 0.071, or 7 points, and a 4-point change is within sampling variability; the claim that support is growing is not supported by the data, although it may be true.

Worked evaluation of a causal claim: "Cities with more parks have lower rates of heart disease; building parks would reduce heart disease." Observational: cities were not assigned parks. Wealthier cities can afford both more parks and better health care, and their residents may differ in age, diet and smoking. The association may be real; the recommendation assumes causation the data cannot show. A defensible report would say "associated with" and note the lurking variables.

Worked evaluation of a graph and a ratio: A report's chart shows a bar for 3.1% and a bar for 3.4% with the vertical axis running from 3.0% to 3.5%, so the second bar is four times as tall, headlined "complaints surge." The absolute change is 0.3 percentage points, a relative increase of about 10%. Redrawn from 0 the bars are nearly equal. The evaluator reports the absolute and relative changes together and asks whether 0.3 points exceeds the year-to-year variation.

Worked evaluation of the passing report: Random selection from the county list, 76% response, 56% ± 4 points. Strengths: population defined, selection random, uncertainty stated, wording available. Remaining concern: if the 24% who did not respond lean one way, the estimate shifts; the report should describe follow-up efforts. Conclusion the data support: a majority of the county's registered voters likely favor the levy, by a margin of a few points.

Represent the checklist: Take one report based on data from a newspaper, website or advertisement. Build a table with the seven checklist questions as rows and two columns: what the report says (quote it) and your evaluation of that link. Mark each row as supported, unsupported or not stated. Finish with one sentence stating the strongest conclusion the data actually support, which may be weaker than the report's.

Independent performance: Find two reports based on data that make different kinds of claims, one an estimate from a survey and one a comparison or causal claim. For each, write an evaluation of about 300 words that (1) names the population and describes the selection, (2) classifies the study type and states whether the conclusion matches it, (3) examines what was measured and how it was worded, (4) checks for a margin of error or a significance judgment and computes an approximate margin yourself if the sample size is given, (5) inspects any graph or ratio for honest presentation, (6) notes who conducted the study, and (7) rewrites the headline as the claim the data support. Success means every judgment points to a specific feature of the report, at least one lurking variable is named for any causal claim, and the rewritten headline is one you could defend with the report's own numbers.