Simpson’s Paradox occurs when a trend within different groups of data exists in one direction, but when the groups are combined, the trend is reversed. This video does a great job of explaining the problem in terms of fruit sellers.
The underlying problem is when there is a “lurking” variable that causes the overall trend in the data, but the data groups are sliced along a different variable. In the fruit example from the video, if you look at each fruit seller’s percentages, then you’d think that Bob is the overall better choice. However, due to the unbalanced way that each seller stocks fruit, it’s could be the case that both fruit sellers have the same supplier. The real pattern in the data, then, is that bananas tend to have a better fresh-to-rotten ratio than apples, not that one seller has a better stock than the other. Also, if you only wanted a piece of fruit and didn’t care if it was a banana or apple, then you’d be much better going to Alice.
Oh, people can come up with statistics to prove anything, Kent.
An actual example of Simpson’s Paradox is the Berkeley Graduate Admissions Gender Bias case from the 1970s. Berkeley was accused of not admitting female applicants at the same rate as male applicants. When examined on the whole, it appeared that the Berkeley graduate school discriminated against women.
However, if you look at each department individually, you find that most of them actually accept women at a higher rate.
Just like we saw in the fruit seller example, there is a lurking variable in the Berkeley admissions data. In this case, the lurking variable is the department to which a prospective student applied. These departments have different acceptance rates, and it turns out that women applied more often to departments with low acceptance rates, which (despite them being more likely to be accepted within most departments) lowered the overall rate of accepted women. The graph below shows the correlation of women tending to apply to departments with lower acceptance rates.
So overall summary statistics don’t always tell the whole story and can obscure multiple underlying trends that interact with each other. It’s always important to go back to the raw data and comb over any conclusions drawn from that data to look for mistakes in the analysis. Unfortunately, sometimes the data isn’t always available. Considering that the Berkeley admissions data is very sensitive, they may have chosen not to release it at a level of granularity where the Simpson’s Paradox was apparent. It goes back to the Transparency vs Privacy dilemma that I’ve talked about before. But as a whole, remaining skeptical of data-drawn conclusions (even your own) will always serve you well.