It’s hard to agree with a study last week that claimed academic medical centers hype their research through their press releases, once you look closely at the process they used to reach that conclusion. The releases are, after all, fundamentally intended to attract reporters’ interest. But the study claims that medical centers go too far in trying to do so.
The researchers’ intent—presumably—was to warn the public of shortcomings in these med-center missives to the media. But instead, they may have muddied the waters by not better understanding the topic and implying that press offices, rather than journalists, are responsible for poor reporting.
Their report, published in the respected Annals of Internal Medicine, concluded: “Press releases from academic medical centers often promote research that has uncertain relevance to human health and do not provide key facts or acknowledge important limitations.”
The researchers looked at press releases issued in 2005 by ten academic medical centers of highest rank and ten of lowest rank according to U.S. News & World Report’s rankings. They analyzed the releases based on nearly three dozen criteria, such as whether or not they included basic details and cautions about the research at hand. Follow-up interviews with information officers at the medical centers provided additional information about the schools’ communications practices.
Predictably, some major news media just parroted the report’s findings without much question and fell into a familiar trap of repeating claims and fixing blame. The Wall Street Journal’s health blog and USA Today’s On Deadline blog seemed to see the report as a way to shift the responsibility for poor medical reporting from the journalists to the PR types. Perhaps they were influenced by Annals of Internal Medicine’s own press release about the study, which totaled only 141 words. Other news media followed suit.
I’ve reported on biomedical research for three decades at one of the country’s largest academic medical centers, so the problems I see with this study—and its conclusions—might be somewhat obvious.
The study argues that many of the centers’ press releases overstated the immediate or future clinical relevance of the work mentioned, frowning especially on the promotion of animal studies. But anyone vaguely familiar with biomedical research knows that animal research is a necessary precursor to human clinical studies. And while it is true that some releases need more warnings about extrapolation to human health, stating what kind of treatments the research could lead to is perfectly legitimate.
More importantly, research at academic medical centers often focuses on the biomechanics that occur at the cellular level, and not always on studies involving new clinical treatments. The logic is to understand the biology at hand in hopes of using it for new findings in the future.
Immediate and assured clinical relevance isn’t a criterion that most scientists would categorically embrace before beginning their research.
Drummond Rennie, former deputy editor at the New England Journal of Medicine, once told Esquire that about half of what was published in that prestigious journal eventually was proven wrong. That is the nature of science and it doesn’t diminish the value of that journal’s offerings.
Other problems with last week’s press-release study abound.
The determination that a particular release was “hyped” was made by two research assistants, designated as “coders,” who evaluated the releases based on a variety of criteria such as the inclusion of key study details (design, size, etc.) and overall clarity. There was a high degree of agreement between coders, but, as the report readily admits, their decisions about whether or not something was exaggerated were subjective. Furthermore, gauging the accuracy of any given release would require ample experience in science/medical reporting, and the report offered no description of the coders’ qualifications for carrying out that task.
For the interview portion of the study, a team member interviewed “the person in charge of media relations” at the medical centers, but it has been my experience that the senior PR official at most institutions has little, if any, hands-on experience in covering research. Better to have discussed the process with the staff science or medical writer if they wanted more accurate answers.
The study pointed out that few of the press releases “provided access to the full scientific report.” But they failed to add that, in many cases, copyright protections prohibit institutions from providing copies of such papers to the news media. The journals publishing these reports hold fast to their legal right that only they can distribute the papers carried in their journals. Omitting that point casts undue blame on the medical centers.
The researchers also faulted some releases for lacking outside review, but that is rarely, if ever, used on such releases, regardless of the subject. Unlike the inherently slow process of journal publication, the rush of daily journalism can hardly accommodate the time needed for outside review. However, at trustworthy communications offices, press releases endure an editing process that is similar to the ones employed in most newsrooms. One or two editors on staff who are experienced in evaluating research with an eye towards news edit the release. It is then sent to the lead researcher, who is asked to check it for technical accuracy alone, not for any editorial slant. No deans, directors, department chairs, or administrators review the story prior to dissemination. This prevents overt PR intent from affecting the story. That’s how it works at our shop, and at a host of other big-time places.
Interestingly, the Annals of Internal Medicine study reported that at all twenty medical centers, it was the individual researchers who requested that their work be covered in a press release, and that at more than half of the centers, administrators made similar requests. There was no suggestion that qualified science or medical writers on staff were functioning as reporters and seeking out newsworthy stories—the practice routinely followed at most major research universities, nor were there any questions regarding this practice in the script (pdf) the authors used in gathering data.
- 1
- 2
Some important observations, Earle.
I'd like to add, as someone who has written stories on press releases for the Associated Press and Business Week, among many others, that reporters are not generally equipped to evaluate the science in journal reports. I've had weeks in which I've written studies on epidemiology, evolutionary biology, genomics, and virology, back to back. No biologist would be equipped to judge such a diverse collection of studies, and reporters can't do it either, even if they happen to have Ph.D.s (which I don't).
The key to this business of covering studies is to find people who can understand and evaluate them, and who are not connected to the research, and to ask them what it means. I've seen many reporters come to believe, after years on the job, that they are experts, when, in fact, they've never examined a patient, run a regression analysis, or analyzed an immune response.
That leads me to one weakness in most press releases. They rarely include that kind of outside, independent assessment. Public information folks rely on their own scientists to evaluate their own research, and that can lead to overstatement of the importance of the study--either because the researchers are caught up in their enthusiasm, or because they hype the results to advance their careers, to promote their university, or to prop up the companies that they own and stand to profit from.
That's where reporters come in; they provide a check on that, if they do their jobs well. Or at least that used to be the case. Now press releases are distributed directly to the public via the web, and the news spreads without any independent assessment.
#1 Posted by Paul Raeburn, CJR on Tue 12 May 2009 at 10:55 PM
Paul is way too humble in his comments. He certainly is one journalist who I've found can effectively deal with journal reports, even across the diverse fields that he mentions. And I would differ, slightly and respectfully, with his suggestion that reporters can't evaluate studies at a fairly basic level. I think they can, given that they evaluate source material for other complicated stories. The kinds of missing elements that I cited in the Annals piece above are fairly basic and are derived from stepping back and looking for holes in the logic -- not a sophisticated understanding of research methodology. I believe that reporters need to be able, and empowered to do just that.
And for the record, I'm not defending all practices concerning all press releases, God forbid! We all have seen disasterous attempts at spinning and messaging. But not all are bad and those dealing with university research tend to be closer to single-source journalism than releases coming from commerce or industry.
Paul's "weakness" of releases that they don't offer "outside, independent assessment" as part of their offering is really puzzling when you consider the role that releases are supposed to play -- that of alerting the news media to a potential story and providing adequate basic information to make an initial determination of viability. If PIOs are supposed to seek outside assessment of their own researchers' work, what does that leave the journalist to do? Moreover, since the folklore in journalism has always been that the provider of a news release is bent on manipulating journalists, then why would journalists trust any supposed "independent assessment" we might offer? Surely, it would be seen as our own spin.
As to the suggestion that researchers overstate to advance their careers, I can only say that in 35 years of doing this work, I've never encountered that. Nor have several colleagues whom I asked that question. Sadly, the assumption is that selfishness is a primary driver of actions by researchers, and while I'm not so naive as to suggest that it doesn't exist, it doesn't dominate the scene.
The lamentation that news "releases are distributed directly to the public via the web, and the news spreads without any independent assessment" is easily fixed, if it is a concern at all. Journalists need to do a better job of evaluating the studies. A bit of time spent at Gary Schwitzer's great website, Health News Review [http://www.healthnewsreview.org/] offers a great primer on what's good and bad about ongoing medical/biomedical reporting.
#2 Posted by Earle Holland, CJR on Wed 13 May 2009 at 10:12 AM