A long standing pet peeve is how the use of figures has been fetishized in political discourse and in our society generally, to the point where many people too easily swayed by argument that invoke data (I discussed this phenomenon at length in the business context in a 2006 article for the Conference Board Review, Managementt’s Great Addiction). And now that what used to be called statistical analysis has now been given mystical powers by calling it “Big Data,” the layperson is going to be subjected to even more torturing of information by people who have incentives to find stories in it, whether those stories are valid or not.
One helpful notion to keep in the back of your mind when looking at studies are basic ideas like: was the sample size large enough? Is is representative? And one always needs to remember that correlation is not causation.
This short video (hat tip Lars Syll) provides some additional insight into the sound use of statistics, which should help in thinking critically about research findings. This video is a couple from 2013 and has lively examples.
If you are interested in more on this topic, I strongly urge you to read the classic and widely-cited 2005, Why Most Published Research Findings Are False. Note that this paper’s warnings apply most strongly to research where the investigator creates his own data set for study, such as medical research. Here is the abstract:
There is increasing concern that most current published research findings are false. The probability that a research claim is true may depend on study power and bias, the number of other studies on the same question, and, importantly, the ratio of true to no relationships among the relationships probed in each scientific field. In this framework, a research finding is less likely to be true when the studies conducted in a field are smaller; when effect sizes are smaller; when there is a greater number and lesser preselection of tested relationships; where there is greater flexibility in designs, definitions, outcomes, and analytical modes; when there is greater financial and other interest and prejudice; and when more teams are involved in a scientific field in chase of statistical significance. Simulations show that for most study designs and settings, it is more likely for a research claim to be false than true. Moreover, for many current scientific fields, claimed research findings may often be simply accurate measures of the prevailing bias. In this essay, I discuss the implications of these problems for the conduct and interpretation of research.
As we know, economics has as different approach in that the discipline relies on mathed-up statements of finding which serves to exclude non-economists from being taken seriously as critics of their work (and as skeptics like Deidre McClosky have pointed out, the parts written up in mathematical form are often the trivial parts of the argument). And they generally prefer abstract models with Panglossian assumptions embedded in them (that economies have a propensity towards equilibrium state at full employment) over empirical work.