Friday, November 6, 2009

The importance of fact-checking

In recent weeks, there has been some debate here in Athens about the safety of raw milk. Two weeks ago, I wrote a letter to the Flagpole weekly, in reply to some fear-mongering on the issue which used false, misleading, and unverifiable statistics.

One of the "facts" I took issue with was the claim that the CDC has reported that from 1998 to 2005, one thousand people fell ill from consuming unpasteurized milk, and two died. As I pointed out in my response, the former figure appears plausible, but the CDC's published statistics offer no indication that two people died from consuming raw milk during that period. The only deaths due to milk consumption the CDC does report are three deaths due to the consumption of pasteurized milk, in 2007 — the epitome of an "inconvenient truth" for those who wish to attack unpasteurized milk.

After some additional research, I found that the claim about 1000 illnesses (actually, the "correct" figure is 1007) and 2 deaths pops up everywhere, including in official documents. Indeed, the FDA offers it in its rather fear-mongering Q&A article on raw milk without offering any attribution other than to mention the CDC.

The FDA's document claims the following: "The Centers for Disease Control and Prevention (CDC) reported that from 1998 to 2005, there were 45 outbreaks of foodborne illness in which unpasteurized milk or cheese likely made from unpasteurized milk were implicated. These outbreaks accounted for 1007 illnesses and 104 hospitalizations, and two deaths. The actual number of illnesses is almost certainly higher, but not all cases recognized are reported."

This claim, or parts of it, is widely copied in other articles (a Google search on the string "1007 illnesses, 104 hospitalizations" gives 884 hits), usually verbatim, and rarely with reference to the original source.

Further digging indicates that the FDA did at one point offer an actual source for the claim, in a March 1, 2007 press release. There, they refer to the March 2, 2007 issue of the CDC's Morbidity and Mortality Weekly Report (MMWR) 56(8):165-167. This report is easily downloadable from the CDC website, and it does not support the FDA's claim. Instead, it notes that over the period 1998-2005, just 33 (not 45!) outbreaks were reported, without specifying anything about number of illnesses or deaths. In turn, the MMWR simply refers readers to the CDC's annual listings which, as I noted, do not indicate any deaths.

Why, do people — and not just random blog posters; the National Environmental Health Association does it too — misleadingly cite this report for a claim which it does not support? Presumably because citing a source makes a claim seem more authoritative, and they cannot be bothered actually to double-check whether that source supports their claim. The FDA surely knew that the citation was false, but probably assumed — correctly — that most people would not double-check.

Still further digging shows that the MMWR of of November 9, 2007 (56(44):1161-1164) does offer the claim the FDA already announced in March. Here we find the statement: "During 1998--2005, a total of 45 outbreaks of foodborne illness were reported to CDC in which unpasteurized milk (or cheese suspected to have been made from unpasteurized milk) was implicated. These outbreaks accounted for 1,007 illnesses, 104 hospitalizations, and two deaths (CDC, unpublished data, 2007)."

Note that they refer to unpublished data (and that the blame is spread to cheese, something most of those who use the claim ignore). Indeed, this data appears never to have been published. That is puzzling, since it obviously contradicts published data (as well as the figure of 33 outbreaks offered in the same publication just a few months earlier). The most likely explanation is that further research showed the data to be unverifiable or faulty. The fact that the claim is offered in an "editorial note" rather than in an actual research report is, perhaps, telling.

Why, then, does the FDA continue to use this claim in its official information? Because it is convenient for the story they wish to tell, one imagines. But their doing so is disingenuous at best, and it permits others to get away with offering misinformation simply by appealing to the "authority" of the FDA, a government agency.

Indeed, numerous online sources simply copy the November MMWR claim, again usually without attribution, down to the mysterious (and utterly unhelpful) citation to "(CDC, unpublished data, 2007)". Even a presumably peer-reviewed article in the journal Food Microbiology (26(6):615-622) gets away with doing so, despite the fact that none of the authors was associated with the CDC, so they obviously had no access to that data, and they do not indicate where they got the figures (probably the FDA press release).

Given how easy it is to access the alleged sources of these claims, it is disappointing how rarely people actually do so. Data offered without a source is unverifiable and hence worthless, so one should always cite sources. Moreover, the point of citing a source is not to look intelligent or well-read; it is to offer your readers the opportunity to learn more and/or to verify that you are right. At a minimum, then, you should make sure that they will actually find what you claim they will.

No comments:

Post a Comment