Sign up for The Media Today, CJR’s daily newsletter.

In January 2024, news outlets in the greater Baltimore area rushed to report on a local scandal out of Pikesville High School, where the principal was placed on administrative leave after an offensive voice recording went viral on Twitter. 

To the thousands of Pikesville residents who shared it—and the millions of users who heard it online—the recording appeared to capture the school’s principal making racist and anti-Semitic remarks about students and staff in a private conversation. But even after the principal denied saying those things, and several local outlets reported that the clip was unverified, many flocked to the suburb to interview parents and other community members, effectively amplifying the allegation and adding fuel to the growing backlash. The principal received death threats online as a result, and the school had to temporarily increase security measures after staff reported feeling unsafe at work.

As the local authorities continued to investigate, however, a crucial fact emerged: the recording had been a fabrication, generated by a disgruntled employee using AI voice cloning. 

“We are relieved to have some closure on the origins of this audio,” Baltimore County Executive Johnny Olszewski said at a news conference on April 25. “However, it is clear that we are also entering a new, deeply concerning frontier.”

Just over a year later, the story serves as a stark illustration of the potential dangers associated with AI-assisted impersonation and blackmail, but it also poses important questions about journalists’ ability—and responsibility—to authenticate potentially damaging content in the AI era, when deceptive “synthetic” media are becoming ubiquitous online.

Demonstrating the technical complexities associated with reliably detecting the presence of AI manipulation, CNN later reported that the Baltimore police had called upon the technical expertise of two forensics experts, as well as FBI consultants, to determine the clip’s origins. This detail begs the question of what, if anything, the local journalists covering the story could have done to independently verify the clip’s veracity (or in this case, its lack thereof). 

Although there is a growing array of tools and paid services available that claim to be able to automatically detect the presence of AI-generated forgeries (a few examples include Deepware Scanner, DeepFake-o-Meter, and Optic), recent studies published by computer scientists and machine-learning experts reveal that the technology underpinning these solutions has largely not kept up with the rapid advance of diffusion models that generate convincing, deceptive content at scale. Some scholars have gone so far as to refer to this area of research as a “battleground” where developers of detection tools are losing more and more ground to the generative-AI industry leaders. 

In an effort to help journalists—particularly those with relatively little technical training—navigate this rapidly changing and often jargon-dense field of research, Tow researchers have scoured much of the recent literature on deepfake detection, and highlighted just a few overarching findings that journalists and newsroom leaders should know about the current state of the art in deepfake detection tech:

Not all types of AI detection are the same—or equal.

Whether there exists a reliable tool for detecting AI manipulation largely depends on the nature of the media being assessed and the kinds of adjustments you’re trying to detect. 

So let’s start with audio. Say a reporter comes across a clip—like the one in question in Pikesville—that they suspect might have been generated using an AI tool. Is there any hope of finding a tool that can reliably (de)authenticate it? The research suggests not.

A 2023 meta-analysis evaluating a variety of methods used to detect manipulation in audio files found that “although most of the existing audio deepfake detection methods have achieved impressive performance in in-domain test[s], their performance drops sharply when dealing with out-of-domain dataset[s] in real-life scenarios.” This is because detection tools are themselves powered by algorithms, which are trained on a necessarily limited number of fake audio files. But when asked to detect a type of content that was generated using a technique that isn’t covered within their training data, they struggle to yield accurate results.

“In other words,” the authors of the 2023 paper write, “the generalization ability of audio deepfake detection systems is still poor.”

Similar patterns can be found in the research related to deepfake videos. 

A 2024 experiment testing a variety of tools designed to detect visual “forgeries” found that different tools employ creative and unique approaches to determining whether a video had been generated or significantly altered using AI, including by analyzing inconsistencies in color, identifying unexpected visual artifacts (or “noise”), detecting unnatural body movements or violations of the laws of physics, and analyzing metadata for clues. However, the authors argue that challenges remain in combining these methods into a single detection tool that can reliably detect manipulation techniques outside of those specifically referenced in its training data.

This is a key failing of detection tools, because as generative-AI software continues to advance and proliferate, it will continue to remain one step ahead of the detection tools.

And although there is some research suggesting that more narrowly focused tools—such as those trained on the voices and faces of specific individuals—might provide a promising avenue for scalable deepfake detection methods in the future, even the most advanced AI detection models struggle with real-world accuracy and generalization at this point.

Detection techniques are easily evaded by those who know how they work.

Furthermore, the authors of the 2024 experiment point out that most of the available detection tools are not well equipped to account for intentional attempts to evade detection on the part of bad actors. “As forgery techniques evolve,” they write, “forged videos may evade detection by introducing interference during the detection process.” 

The authors of another paper, published in April 2023, make a similar argument by demonstrating that deepfake creators can make specific visual or statistical adjustments to a fake image or video that are more likely to help it fly under the radar of detection tools. These adjustments include using image filters that smooth out unnatural textures and change the lighting of an image, or manually removing visual inconsistencies and anomalies that the detector might pick up on. 

This research reveals that there are multiple ways to make a synthetic image look real, including those that cannot be identified by most of the available deepfake detection tools on the market today. From a journalistic perspective, this means that well-resourced malicious actors—including foreign governments—have the ability to evade detection from even the most advanced detection methods.

Even in cases where detection is possible, interpretation of detection results remains a challenge.

In February 2024, WITNESS researchers shirin anlen and Raquel Vázquez Llorente directly compared the reliability of a variety of deepfake detector tools popular among journalists. The software on their list included Optic, Hive Moderation, V7, Invid, and Deepware Scanner, as well as others. The researchers found that while these tools can serve as a “great starting point” for a more comprehensive verification process, particularly when paired with other open-source image verification techniques, their results can be difficult to interpret.

This is because, as the authors write, “most of the results provided by AI detection tools give either a confidence interval or probabilistic determination (e.g., 85% human), whereas others only give a binary ‘yes/no’ result. It can be challenging to interpret these results without knowing more about the detection model, such as what it was trained to detect, the dataset used for training, and when it was last updated.”

If a journalist uploads a dubious image to a detection tool, and the tool says that the photo is “70 percent human” and “30 percent artificial,” this tells the reporter very little about which of the image’s elements were digitally altered, and even less about its overall veracity. Labeling an image as even partially “artificial” implies (potentially falsely) that the picture has been significantly changed from its original state, but says nothing about how the nature, meaning, or implications of the image may have changed.

Furthermore, previous research suggests that labeling any content as even partially manipulated has significantly detrimental effects to its perceived veracity, meaning that a hypothetical reporter might assume that a photo that had simply been color-corrected using AI-assisted photo-editing software was entirely “fake,” if the detector labeled it as partially “artificial.”

Detection tools can contribute to a false sense of security. 

A 2024 experiment conducted at the University of Mississippi demonstrated that journalists with access to deepfake detection tools sometimes overrelied on them when attempting to verify potentially synthetic videos, especially when the tools’ results aligned with their initial instincts, or when other attempts to verify the suspicious videos proved inconclusive. 

“This signals the need for cautions around deepfake detector deployment and highlights the importance of improving explainability,” the authors explain.

Even in cases where the journalists didn’t take the detection tools at face value, their results often contributed to doubt and confusion. “We observed that often the results of the detection tools added to the participant’s uncertainty if its results contradicted their other verification steps,” the study reads. “Sometimes a response from the detector could affect their perception of subsequent and even previous actions, and they could become suspicious of other characters in the scenario.”

Ultimately, these results suggest that blind faith in AI detection can weaken journalistic rigor and could lead to the spread of false or misleading information.

The Takeaway for Newsrooms

These studies make one thing clear: deepfake detection tools cannot be trusted to reliably catch AI-generated or -manipulated content. While they are often marketed as solutions to the growing problem of deceptive synthetic media, the reality is far more complicated. Most struggle with generalization, meaning they fail when confronted with deepfakes generated using new techniques. Many produce ambiguous or misleading results, which can cause more confusion than clarity. And even worse, detection tools are not designed to account for bad actors who deliberately manipulate synthetic media in an effort to evade detection.

Overreliance on these tools doesn’t just risk misclassification—it can actively mislead journalists into making incorrect editorial decisions. False negatives may allow sophisticated deepfakes to slip through undetected, while false positives could lead to the unnecessary dismissal of legitimate content. 

That said, deepfake detection tools can have some utility—if used with a high degree of skepticism. Machine-learning experts, including many cited here, argue that these detectors should be seen as one tool among many, rather than a definitive source of reliable answers. When interpreted cautiously and integrated into a broader verification strategy, they may assist journalists in navigating the increasingly deceptive landscape of AI-generated media.

Has America ever needed a media defender more than now? Help us by joining CJR today.