Amidst the current batch of child safety bills in Congress, a familiar name appears: the STOP CSAM Act. It was previously introduced in 2023, when I wrote about the threat the bill posed to the availability of strong encryption and consequently to digital privacy in the post-Roe v. Wade era. Those problems endure in the 2025 version (which has passed out of committee), as explained by the EFF, the Internet Society, and many more civil society orgs. To their points, I’ll just add that following the Salt Typhoon hack, no politician in any party has any business ever again introducing a bill that in any way disincentivizes encryption.
With all that said, the encryption angle is not the only thing worth discussing about the reintroduced bill. In this post, I’d like to focus on some other parts of STOP CSAM – specifically, how the bill addresses online platforms’ removal and reporting of child sex abuse material (CSAM), including new language concerning AI-generated CSAM. The bill would make platforms indicate whether reported content is AI – something my latest research finds platforms are not all consistently doing. However, the language of the requirement is overbroad, going well beyond generative AI. What’s more, forcing platforms to indicate whether content is real or AI overlooks the human toll of making that evaluation, risks punishing platforms for inevitable mistakes, and assumes too much about the existence, reliability, and availability of technological tools for synthetic content provenance detection.
STOP CSAM Would Make Platforms Report Whether Content Is AI-Generated
One of the many things the STOP CSAM bill would do is amend the existing federal statute that requires platforms to report apparent CSAM on their services to the CyberTipline operated by the nonprofit clearinghouse the National Center for Missing and Exploited Children (NCMEC). The 2025 version of the bill dictates several new requirements to platforms for how to fill out CyberTipline reports. One is that, “to the extent the information is within the custody or control of a provider,” every CyberTipline report “shall include, to the extent that it is applicable and reasonably available,” “an indication as to whether” each item of reported content “is created in whole or in part through the use of software, machine learning, artificial intelligence, or any other computer-generated or technological means, including by adapting, modifying, manipulating, or altering an authentic visual depiction” (i.e., real abuse material). If a platform knowingly omits that information when it’s “reasonably available” – or knowingly submits a report that “contains materially false or fraudulent information” – STOP CSAM permits the federal government to impose a civil penalty of $50,000 to $250,000.
This provision is pertinent to the findings of a paper about AI-generated CSAM that my colleague Shelby Grossman and I published at the end of May. Based on our interviews with platforms (including some AI companies), we find that platforms are generally confident in their ability to detect AI CSAM, and they’re reporting AI CSAM to the CyberTipline (as they must), but it appears platforms aren’t all consistently and accurately labeling the content as being AI-generated when submitting the CyberTipline reporting form (which includes a checkbox marked “Generative AI”). When we interviewed NCMEC employees as part of our research, they confirmed to us that they receive CyberTipline reports with AI-generated files that aren’t labeled as AI. Our paper urges platforms to (1) invest resources in assessing whether newly identified CSAM is AI-generated and accurately labeling AI CSAM in CyberTipline reports, and (2) communicate to NCMEC the platform’s policy for assessing whether CSAM is AI-generated and labeling it as such in its reports.
In short, current practice for AI CSAM seems to be to remove it and report it to NCMEC, but our sense is that most platforms are not prioritizing labeling CSAM as AI-generated in CyberTipline reports. Presently, reporting CSAM (irrespective of whether it’s AI or real) is mandatory, but the statute doesn’t give that many specifics about what information must be included, meaning most parts of the CyberTipline reporting form are optional. Thus there’s currently no incentive to spend extra time trying to figure out whether an image is AI and checking another box (all while the neverending moderation queue keeps piling up). STOP CSAM would change that, and would likely lead platforms to spend more time filling out CyberTipline reports about the content they’d quickly remove.
The $250,000 question is: How accurate does an “indication as to whether” a reported file is partially/wholly AI-generated have to be – and how much effort do platforms have to put into it? Can platforms rely on a facial assessment by a front-line content moderator, or is some more intensive analysis required? At what point is information about a file not “reasonably available” to the platform, even if it’s technically within the platform’s “custody or control”? Also, a lot of CyberTipline reports are submitted automatically without human review at the platform, typically where a platform’s CSAM detection system flags a hash match to known imagery that’s been confirmed as CSAM. How would this AI “indication” requirement interact with automated reporting?
The Reporting Requirement Goes Beyond “AI”
STOP CSAM’s new reporting provision doesn’t require the reporting only of AI-generated imagery. Read the language again: when submitting a CyberTipline report, platforms must include “an indication as to whether the apparent [CSAM] is created in whole or in part through the use of software, machine learning, artificial intelligence, or any other computer-generated or technological means, including by adapting, modifying, manipulating, or altering an authentic visual depiction.”
That goes well beyond the “Generative AI” checkbox currently included in the reporting form (which can already mean multiple different things if it’s checked, according to our interview with NCMEC). Indeed, this language is so broad that it seems like it would apply even to very minor changes to real abuse images, like enhancing the brightness and saturation of the colors, or flipping it so it’s a mirror-image. I’m not sure why or how a platform could reasonably be expected to know what edits have been made to an image. Plus, it’s strange to equate a fully AI-generated image with a real image that’s merely had the color saturation tweaked in a photo editing app. Yet the bill language treats those two things as the same.
This broad language would turn that “Generative AI” checkbox into a catch-all. Checking the checkbox could equally likely mean (1) “this is a digital image of a child who’s actively being abused which has been converted from color to grayscale,” (2) “this is an image from a years-old known abuse image series that’s been altered with Photoshop,” (3) “this is a morphed image of a real kid that’s been spit out by an AI-powered nudify app,” or (4) “this is a fully virtual image of an imaginary child who does not exist.” How is that useful to anyone? Until NCMEC adds more granularity to the reporting form, how is NCMEC, or law enforcement, supposed to triage all the reports with the “Generative AI” box checked? Is Congress’s expectation that platforms must also include additional details elsewhere (i.e. the free text entry box also included in the CyberTipline form)? Will they be fined if they don’t?
It’s not a speculative concern that platforms would comply with STOP CSAM by reporting that an image has an AI element even if it merely has minor edits. In both this AI CSAM paper and our previous paper on the CyberTipline, we found that platforms are incentivized to “kick the can down the road” when reporting and let NCMEC and law enforcement sort it out. As one platform employee told us, “All companies are reporting everything to NCMEC for fear of missing something.” The burden then falls to NCMEC and law enforcement to deal with the deluge of reports of highly variable quality. Congress reinforces this incentive to over-report whenever it ups the ante for platforms by threatening to punish them more for not complying with increased reporting requirements – such as by fining them up to a quarter of a million dollars for omitting information that was “reasonably available.” The full Senate should keep that in mind should the bill ever be brought to the floor.
The Human Cost of the “Real or AI?” Determination
Although our report urges platforms to try harder to indicate in CyberTipline reports whether content is AI-generated, there are downsides if Congress forces platforms to do so. In adding that mandate to platforms’ CyberTipline reporting requirements, the STOP CSAM bill does not seem to contemplate the human factors involved in making the call as to whether particular content is AI-generated.
As our paper discusses, there are valid reasons why platforms might hesitate to make the assessment that a file is AI-generated or convey that in a CyberTipline report. For one, platforms may not want to make moderators spend additional time scrutinizing deeply disturbing images or videos. Doing content moderation for CSAM was already psychologically harmful work even before generative AI, and we heard from respondents that AI-generated CSAM tends to be more violent or extreme than other material. One platform employee memorably called it “nightmarescape” content: “It’s images out of nightmares now, and they’re hyperrealistic.” By requiring an indication of whether reported content is AI, the STOP CSAM Act would incentivize platforms to make moderators spend longer analyzing content that’s particularly traumatic to view. Congress should not ignore the human toll of their child-safety bill.
Platforms may also fear making the wrong call: What if a platform reports an image as AI CSAM when it’s actually of a real child in need of rescue? What if the law enforcement officer who receives that report deprioritizes it for action out of the mistaken belief that it’s “just” AI, thereby letting the harm continue? Besides the weight of that mistake on platform personnel’s conscience, there’s also the specter of potential corporate liability for the error. (Platforms are supposed to be immune from liability for their CyberTipline reports, but that isn’t always the case.)
STOP CSAM would exacerbate the fear of getting the “real or AI?” assessment wrong. Platforms could incur stiff fines if a CyberTipline report knowingly omits required information or knowingly includes “materially false or fraudulent” information. That is, a platform could get fined both for failing to indicate that content is AI-generated when in fact it is, and for wrongly indicating that it is when in fact it isn’t, if the government concludes the conduct was knowing. (Even if the platform ends up getting absolved, the path to reaching that outcome will likely be costly and intrusive.)
Forcing platforms to make this assessment, while threatening to fine them for getting it wrong, could improve the consistency and accuracy of platforms’ CyberTipline reporting for AI-generated content. But it won’t come without a human cost, and it won’t guarantee 100% accuracy. There will inevitably be errors where real abuse imagery is mistakenly indicated to be AI (potentially delaying a child’s rescue), or where, as now, AI imagery is mistakenly indicated to be real (potentially wasting investigators’ time).
To try to comply while mitigating their potential liability for errors, platforms might submit more CyberTipline reports with that “Generative AI” box checked, but add a disclaimer: that this is the platform’s best guess based on reasonably available information, but the platform is not guaranteeing the assessment’s accuracy and the assessment should not be relied on for legal purposes, etc. If platforms hedge their bets, what’s the point of making them check the box?
What’s the State of the Art for AI CSAM Detection?
Congress seems to believe that platforms know for a fact whether any given image they encounter is AI-generated or not, or at least that they can conclusively determine the ground truth. I’m not sure that’s true yet, based on our interviews for the AI CSAM paper.
A respondent from a company that does include AI labels in its CyberTipline reports told us that they still use a manual process of determining whether CSAM is AI-generated. For now, most of our respondents believe the AI CSAM they’re seeing still has obvious tells that it’s synthetic. But moderators will need new strategies as AI CSAM becomes increasingly photorealistic. Already, one platform employee said that even with significant effort, it remains extremely difficult to determine whether AI-generated CSAM is entirely synthetic or based on the likeness of a real child.
When it comes to content provenance, Congress should take care not to impose reporting requirements without understanding the current state of the technology for detecting AI content as well as the availability of such tools. True, there are already hash lists for AI CSAM that platforms are implementing, and tools do exist for AI CSAM detection. One respondent said that general AI-detection models are often sufficient to determine whether CSAM is AI-generated; we heard from a couple of respondents that existing machine learning classifiers do decently well at detecting AI CSAM, about as well as they do at detecting traditional CSAM. However, we also heard that the results vary by tool and tend to decline when the AI content is less photorealistic. And even currently performant tools can’t remain static, since the cat-and-mouse game of content generation and detection will continue as long as determined bad actors keep exploiting advances in generative AI.
There’s also the issue of scale. Congress shouldn’t expect every entity that reports CSAM to NCMEC to have the same resources as a massive tech company that submits hundreds of thousands of CyberTipline reports annually. Implementing AI CSAM detection tools might not be appropriate for a small platform that submits only a handful of reports each year and does everything manually. This goes back to the question of how much effort a platform must put into indicating whether reported material is AI, and how accurate that indication is expected to be. Even for big platforms, it is a challenge to determine conclusively whether highly realistic-looking material is real or AI-generated, much less for small ones. Congress should not lose sight of that.
Conclusion
The reboot of STOP CSAM is just one of several bills introduced in this Congress that involve AI and child safety, of which the TAKE IT DOWN Act is the most prominent. Having devoted most of my work over the past two years to the topic of AI-generated CSAM, it is gratifying to see Congress pay it so much attention. That said, it’s dismaying when legislators’ alleged concern about child sex abuse manifests as yet another plan to punish online platforms unless they “do better,” without reckoning with the counterproductive incentives that creates, the resources available for compliance (especially to different-size platforms), or the technological state of the art. In that regard, unfortunately, the new version of STOP CSAM is the same as the old.
Riana Pfefferkorn (writing in her personal capacity) is a Policy Fellow at the Stanford Institute for Human-Centered AI.