Texas Sharpshooter Fallacy

09 Feb 2015

Ice Cream Crimes

The Fallacy

Exterior. Texas Farm. A young cowboy aims his pistol at the side of a barn and fires several shots into the side of it. He then walks over to the barn and paints a target where most of the bullet holes were concentrated, claiming himself to be quite the sharpshooter.

The above illustration points out the inherent problem with an after-the-fact analysis of data, called the Texas Sharpshooter Fallacy. This fallacy is closely related to the statistical fallacy called the multiple comparisons problem, where you can do what seems like a reasonably scientific study, but instead of testing one small relationship, you cast too wide a net in your hypothesis and point to a causal effect when one aspect of the hypothesis is met retrospectively.

The Good (Intentions)

These fallacies are what led a well-intentioned group of concerned individuals (and some scientists who should have known better) to claim that electric lines were giving their kids cancer. They reached this conclusion despite scientists telling them that the “fields [from the power lines] are 1/200th or so of the earth’s magnetic fields” and having the “45,000-member American Physical Society…release a report last month saying that cancer fears were ‘unfounded.’” The evidence supporting this link between power lines and cancer were not sound enough to warrant any level of panic, but the misinformed group of concerned individuals insisted that what they were seeing was more than just statistically insignificant noise.

The Bad (Data)

Some concerned parents mapped out the concentration of childhood luekemia in 4 zip codes in their area, pointing to the raised level of occurrances as proof. Sound like a familiar technique? The childhood luekemia data points are like the “sharpshooter’s” bullet holes. Cases of the disease won’t be evenly distributed geographically, so a concentration of occurrences in 4 zip codes is not very statistically unlikely. This method of analysis ignores other data entirely. For instance, a “densely populated ZIP code a few miles away has just five cancer cases, less than half the expected number, and it, too, is criss-crossed with power lines.”

Another piece of evidence that the proponents of the electricity-cancer link used was a Swedish study that looked at “everyone living within 300 meters of Sweden’s high-voltage transmission line system over a 25-year period.” This study did find 4 times the number of childhood luekemia cases in this population of Swedes. However, their study looked at around 800 other possible ailments. Selecting a single result from that study is a textbook case of the multiple comparisons problem.

The Ugly (Cost)

While the arguments for the cancer link seem reasonable on the surface, it’s important for data-minded people to spot these fallacies when they arise. Also, it’s important to educate others as well, since having any proportion of society succumb to these fallacies can be costly. When the article on this controversy was published in 1995, an estimated $1 billion per year was being spent on this debate. That’s a lot of ice cream.

comments powered by Disqus