Sign up for The Media Today, CJR’s daily newsletter.
DETROIT, MI — Josh Mandel didn’t just lose his 2012 bid to unseat Sherrod Brown and claim one of Ohio’s spots in the US Senate. The state treasurer also took home the “Pants on Fire” crown for his putrid ratings on PolitiFact Ohio’s “Truth-O-Meter.”
Statewide campaigns are heating up again in Ohio this year, but it’s unclear if a Pants on Fire “winner” will be crowned: the Cleveland Plain Dealer, the state’s largest paper, has ended its partnership with PolitiFact.com. The Plain Dealer and its digital affiliate, though, are very much still in the factchecking business. Say goodbye to the Truth-O-Meter; say hello to “Truth in Numbers.”
The variations between the two models point to real differences of approach within the booming factchecking movement, though both Truth in Numbers and the Truth-O-Meter have faced some criticism for how they attempt to attract readers’ attention.
But for the Cleveland news outlet, going it alone was a business decision as much as an editorial one—and the move may reflect a broader tension, in a media ecosystem focused on web traffic, between seeking partnerships and building your own digital brand online.
News that The Plain Dealer and its online sibling, Northeast Ohio Media Group, would be parting ways with PolitiFact came in a January column by reader representative Ted Diadiun. Diadiun devoted much of the column to criticizing the Truth-O-Meter, which grades politicians’ statements on a scale from “true” to “pants on fire,” as a “gimmick” that drew attention away from the research and analysis found in PolitiFact posts.
But those complaints “did not figure in the decision” to part company, he wrote. And in any case, not everyone in Cleveland shares Diadiun’s perspective.
“I loved our experience with PolitiFact,” said Chris Quinn, vice president of content at NEOMG and formerly an assistant managing editor at The Plain Dealer, where he participated in Truth-O-Meter rulings. Quinn highlighted the transparency and cross-market collaboration of PolitiFact’s model—and even the “the eye-catching gadgetry of the Truth-O-Meter.” (NEOMG was created simultaneously with layoffs at The Plain Dealer last summer; both the print and digital companies contribute to Cleveland.com and to the print newspaper.)
In fact, Quinn credits PolitiFact Ohio and those Truth-O-Meter ratings with a major role in the 2012 Senate campaign. “Mandel received so many ‘false’ and ‘pants on fire’ ratings that the ratings themselves became news and focused a statewide spotlight on his inability or unwillingness to tell the truth,” he said. “He did not like that attention and eventually altered his campaign, reducing the number of false statements.” (Some examples of the Ohio media focus on Mandel’s accuracy are here, here, and here; there was also a counter-narrative in conservative media that challenged PolitiFact Ohio’s non-partisan bonafides.)
Editors in Cleveland sometimes did struggle to choose a rating, Quinn acknowledged, and that did affect their approach. “We ultimately learned what statements worked best with the Truth-O-Meter and steered our rulings to those, leaving other statements off the table, not to be factchecked.”
But the main concern, he said, was about business strategy. “Ultimately, what did not work for us was putting the fruits of our labor on another web site”—that is, on PolitiFact.com.* “We want visiting our site to be part of the DNA of Northeast Ohio,” Quinn said. “With our increased investment in the digital space last August, it no longer made sense to have our work appear on any other site.”
The PolitiFact Ohio page is still up, and PolitiFact is seeking a new “on-the-ground partner” in the state to factcheck state and local government, said editor Angie Drobnic Holan. As for the “gimmicky” nature of the Truth-o-Meter, she said, the approach is “fun and easy to understand. Readers quickly grasp our straightforward ratings system. Our full reports and online source lists provide them with details, nuance, and transparency.” She declined to say how much The Plain Dealer had paid to license the PolitiFact franchise; in general, “costs depend on the size of the news organization, the specific market, and other details,” she said. (Disclosure: PolitiFact’s new PunditFact project is supported by the Democracy Fund, which is also a major supporter of the United States Project.)
Meanwhile, readers in Ohio have now seen the first couple installments of Truth in Numbers, the new approach to factchecking at The Plain Dealer/NEOMG. Appearing less frequently than PolitiFact posts, TiN differs in a few notable ways—and offers what might be called a gimmick of its own.
TiN posts assess the entirety of a political communication, rather than just a single statement, and they attempt to measure fairness and substance in addition to accuracy. A 10-point scale is meant to bring transparency to the judgment calls:
- Are the words literally accurate? (0-3 points)
- Is the message accurate? (0-3 points)
- Does the message have substance? (0-2 points)
- Is the message fair? (0-2 points)
As for the new gimmick: the subject of a factcheck is featured on Cleveland.com a few days before the post itself, and readers are invited to offer their own ratings. Both the reader and staff evaluations are featured in the final column, along with the reporting and analysis behind the staff conclusions. On the limited evidence to date, readers grade much more harshly than journalists.
The refurbished approach drew mixed reviews from a pair of scholars who have written about factchecking for CJR.
“The scoring system seems a bit confusing, but it’s an interesting idea to try to separate ‘accuracy’ from ‘fairness’ in the analysis,” said Lucas Graves, an assistant professor in journalism at the University of Wisconsin. “It helps to underscore that a claim can be technically accurate but also deeply misleading.” In fact, this is the shift that Quinn says journalists in Cleveland are happiest with so far: “People can speak the literal truth while being misleading as hell, and we think this system catches that.”
And Graves is a fan of the reader rankings—he called it “a great idea, mainly as a way to show how tricky these judgments can be.”
Brendan Nyhan, a political scientist at Dartmouth College and now a contributor to The Upshot, had a more tempered evaluation.
Nyhan’s analysis of the PolitFact’s Truth-O-Meter echoes The Plain Dealer’s caution: While the scale “has clearly been an important factor in their success,” he said, the subjectivity required in assigning a rating can create problems. (He wrote about these concerns here and here.)
Importantly, the Truth in Numbers model shifts away from true/false ratings, which, Nyhan said, is often not a useful way to think about political statements: again, technically true or unfalsifiable statements may also be unsupported by the best available evidence, or constructed in a way that creates a false impression. But the scale still introduces subjectivity to the ratings process, he pointed out: What does a message have to do in order to be “fair” or have “substance”?
As for the reader ratings, Nyhan said he is “dubious that this approach will provide useful information.” While TiN is still new and deserves time to develop its audience, he said, it will always be an unrepresentative group that participates in the process, and those readers are likely to be heavily influenced by their political beliefs. “It’s fine if they want to offer the option for readers to rate statements as a comments-type feature, I suppose, but I see no purpose in featuring the results in their coverage.”
Quinn’s view is that the system could actually cool the anger of politically motivated readers, rather than bait it. “Asking the public to help us do the ratings provides a vent to those who become angry when they think we blow it.”
As for Diadiun, the reader rep, he gave Truth in Numbers a thumbs-up in a recent column—though he admonished readers to “take a certain amount of responsibility” and avoid “blind partisanship” in their ratings.
To help prevent trolling, anyone who participates in the ratings must sign in using a Cleveland.com account. Those accounts are anonymous, but the log-in gives the paper a somewhat better ability to track zealots who give perfect scores to everyone in one party and zeroes to people in the other party. “Ultimately, we might weed those scorers out of the public ratings,” Quinn said. “We might also go with median scores instead of averages if we find that is more fair. This is a bit of an experiment, and we do expect to tweak it.”
However the Truth in Numbers experiment plays out, it’s good to see The Plain Dealer and NEOMG staying on the truth squad beat in the wake of last year’s layoffs. And with the factchecking movement still young, there’s room for some experimentation with the form. In the long run, though, the stakes for finding the most effective model are high. Every election year, huge amounts of money and round-the-clock political advertising dominate the state.
The value of an informed electorate means that “factchecking in Ohio is about as important a duty as I can imagine for a news organization,” Quinn said. “Doing it is part of our deal with this community to be a proper watchdog. It makes a difference.”
* The original version of this sentence incorrectly said that PolitiFact Ohio posts did not run on Cleveland.com. In fact, they appeared at both Cleveland.com and PolitiFact.com. CJR regrets the error.
Has America ever needed a media defender more than now? Help us by joining CJR today.
