Sign up for The Media Today, CJR’s daily newsletter.
Contrary to what you might read on social media, and although we may read the news through blue- or red-tinted lenses, Democrats and Republicans continue to trust traditional media sources over sites that cheerlead for their teams or peddle fiction friendly to them.
“Despite substantial partisan differences, we find that laypeople across the political spectrum rated mainstream sources as far more trustworthy than either hyper-partisan or fake news sources,” write Gordon Pennycook, a professor at the Hill/Levene Schools of Business at the University of Regina in Saskatchewan, and David G. Rand, a professor at the Massachusetts Institute of Technology’s Sloan School, in “Fighting misinformation on social media using crowdsourced judgments of news source quality.” Pennycook and Rand, who drew on two polls to reach their conclusions, propose using the crowd’s collective sense of trust and distrust as a deciding factor, to elevate more credible sites and push less credible sites down.
Pennycook and Rand conducted two surveys of about 1,000 Americans each. The first went out in January to volunteers to a survey request posted at the Amazon Mechanical Turk piecework portal, then another in August via the Lucid survey service, an online pool of research participants who were “nationally representative on age, gender, ethnicity, and geography.” Both platforms paid respondents trivial sums for their time.
Both surveys asked respondents to state their familiarity or lack thereof with these sites, drawn from lists of high-traffic news sites as well as those of hyper-partisan and fake-news sources compiled by such sources as BuzzFeed and PolitiFact. They then requested respondents to rank their trust in these sites (presented in a randomized order) on a scale from 1 to 5, after which the authors calculated those scores as a percentage of a maximum-trust figure of 1.0. Each poll also asked respondents about their political leanings, 2016 voting history, and basic demographic data before requesting them to answer some basic math problems as a test of cognitive skills.
Both surveys found higher trust among self-identified Democrats and Republicans for mainstream news sources—from The New York Times to MSNBC to The Wall Street Journal to Fox News—than for hyper-partisan sites, such as the right-wing publication Daily Caller and the left-wing group blog Daily Kos, as well as sites the authors identified as fake news vendors, such as the Conservative Daily Post and Now 8 News. The authors did not file the notoriously phony InfoWars in the fake-news category, which Rand says was a result of two of three lists categorizing it as hyper-partisan. These surveys did not differentiate between the credibility of news and opinion sections—a fault line at the Journal and Fox.
For example, Republicans in the second survey gave CNN a .34 trust rating, but their faith in President Trump’s least favorite news network still exceeded that placed in Breitbart News, which earned a .27 trust rating—a gap Rand said was statistically significant. Similarly, Democrats trusted Fox News (which received a .34 rating) more than Crooks and Liars (.19), which specializes in video clips of Republicans making dubious statements.
Average trust ratings for each source among Democrats and Republicans in the Lucid survey.
This data also showed Republicans’ trust scores fell significantly below the trust percentages of a panel of professional fact-checkers, recruited from the Poynter International Fact-Check Network to provide a reality check on the crowdsourced assessment. Democrats’ scores did not feature such a gap.
“Neither party trusted the unreliable outlets,” Rand observes. “The problem for Republicans was that they did not trust reliable outlets as much as the Dems.” The lone exception is Fox, which earned a .55 rating among GOPers.
Average trust ratings by professional fact checkers in the Lucid survey. (Click for a larger image)
Pennycook and Rand’s paper also includes troubling findings about people’s willingness to acclimate themselves to placing trust in the wrong places. To wit, lack of familiarity with a site yielded uninformed trust scores that roughly lined up with the judgments of fact-checkers, while spending time at a site led many people to inform themselves into an incorrect judgment.
Both surveys broke out trust scores from respondents who said they recognized the sites in question–and the roughly 15 and 9 percent who declared themselves acquainted with the hyper-partisan and fake-news sites trusted those sites a great deal more. Hyper-partisan sites, for example, averaged a .45 trust score among Republicans familiar with them, versus .4 for mainstream sites; among Democrats, the figures were a less-scary .38 for hyper-partisan sites and .57 for mainstream news.
Average trust ratings for each source when restricting to familiarity ratings in the Lucid survey.
With those issues in mind, Pennycook and Rand propose using these crowdsourced trust scores “in a non-linear concave fashion”—that is, punish the least-trusted sites with low visibility in news feeds, letting sites that pass a threshold of perceived credibility take up the news-feed real estate yanked from the disfavored sites.
In other words, Facebook’s Trusted Sources initiative, announced by the company a year ago, isn’t as crazy as some critics said at the time. The initiative, in which Facebook would survey its users to determine the sites they trust and highlight those sites in the News Feed, was a direct inspiration for the study, which Rand sees as “a king of policy evaluation for that plan.”
“At the time, everyone seemed to think it was a terrible idea, because they thought FB users would be horribly biased in their news source evaluations,” wrote Rand. Instead, he says, these studies show Facebook is “on to something promising!”
Facebook, however, has said next to nothing about the Trusted Sources effort since its launch last January. It hasn’t published the trust scores that resulted from its surveys or described the effects of its subsequent up-rankings beyond such bland verbiage as a September post reporting that “broadly trusted news organizations do get a boost within News Feed.”
Facebook did not respond to a request for comment. A study released in March by NewsWhip, however, found that “misinformation” and phony news continued to do well.
Rand admitted that only Facebook can conclusively answer this by documenting what it’s been doing to people’s News Feeds. But he added that he doesn’t expect any such transparency: “The issue with Facebook is always that we never really have any idea of what they are actually doing.”
Has America ever needed a media defender more than now? Help us by joining CJR today.
