Thursday, October 14, 2010

Medical studies show...

Excellent article in the November 2010 issue of the Atlantic Monthly: Lies, Damned Lies, and Medical Science, by David H. Freedman. The title is an obvious reference to "lies, damned lies, and statistics", a popular indictment of the way statistics can be used to deceive as easily as they can be used to inform. (Side note: I always thought it was a quotation from Disraeli, but Wikipedia informs me that this attribution is faulty; apparently Mark Twain introduced the attribution back in 1906 but without any evidence to back it up).

The Atlantic article is a profile of Dr. John Ioannidis, who has made a career of pointing out the low quality of published medical research. He has hypothesized that about 80% of non-randomized studies are later refuted, as well as 25% of studies involving randomized trials, and even up to 10% of large-scale randomized trials. The article does a fine job of discussing the myriad ways in which the structure of the field and its professional incentives make such high levels of faulty research likely: "Simply put, if you’re attracted to ideas that have a good chance of being wrong, and if you’re motivated to prove them right, and if you have a little wiggle room in how you assemble the evidence, you’ll probably succeed in proving wrong theories right."

(Although it is not one of the main points, Freedman also notes that many published findings may be not so much wrong as spurious: if you take a large enough sample, you will inevitably find some patterns that appear meaningful even though they are not.)

A key message in the article is that the medical research community is not surprised by Ioannidis's findings, but that faulty research continues to have an impact on medical treatment and on public perceptions. One obvious conclusion from the article is that the incentive structure in medical science (the need to have interesting findings, the need to obtain funding from sources that introduce a conflict of interest into one's work) is deeply flawed and should be overhauled if possible. This is not a new argument, but Ioannidis' work gives us added ammunition for making it.

Another possible conclusion is that somehow the medical community is being disingenuous and/or deceptive, because although they are not surprised by the findings, they continue to publish faulty studies, and do flawed research. But I think this conclusion would be unfair. I don't think anyone who has thought about the pressures of publishing in science and about the scientific method (i.e. a much larger population than just medical researchers) would be particularly surprised by Ioannidis' findings.

Indeed, I would argue the situation is cause for concern at least as much because of what it tells us about the lack of understanding of the scientific method and the publication process among the media that transmit the findings and among the general public that 'consumes' them. Improved statistical literacy among reporters and the public alike would go a long way towards making Ioannidis' findings a lot less shocking. Yet one more reason to think Darrell Huff's charming classic How to Lie with Statistics ought to be required reading in high school.

No comments:

Post a Comment