Hugging Face deepfakes report finds weak controls on image models
AI Forensics says popular Hugging Face image tools accepted prompts for nonconsensual sexual deepfakes, including requests involving children.
By Hana Yoshida · Markets Reporter
3 min read
A new AI Forensics report says Hugging Face deepfakes can be made through popular image-editing models hosted on the platform with little resistance from safeguards. The European nonprofit said seven of the nine most-used image-editing models it tested complied with a plain request to remove clothing from women in images.
The findings put pressure on Hugging Face, a major repository for open-source AI models, over how it polices tools that can generate nonconsensual intimate imagery. AI Forensics said the issue appears to sit at the platform level because the hosted models it examined did not block basic sexualized editing prompts.
What did AI Forensics find on Hugging Face?
AI Forensics said it tested top image-editing models on Hugging Face using the same direct prompt for each one, without trying to evade filters through coded language or indirect wording. The nonprofit contrasted that with mainstream AI products such as Google’s Gemini and OpenAI’s ChatGPT, which it said generally use guardrails to refuse prompts that undress or sexualize people.
The group also set up test image-editing Spaces on Hugging Face to observe what users would submit. In this context, the report used Spaces to refer to image-editing areas hosted on Hugging Face that receive prompts and images from users.
AI Forensics said those test Spaces were designed so they would not generate the requested images. Even so, over seven days they received more than 1,000 prompts and images, according to the nonprofit.
The group said 73 percent of the submissions were sexual. Among the sexual requests, 83 percent sought to undress a person in an image, and 95 percent of those targeted women, according to AI Forensics. Nearly 7 percent of the sexual requests were directed at children, the nonprofit said.
What does Hugging Face policy say?
Hugging Face’s content policy bars harmful or abusive content, including sexual material made without explicit consent and underage nudity. AI Forensics said its findings show that those rules are not being enforced strongly enough across image-generation and image-editing tools hosted on the platform.
Paul Bouchaud, a lead researcher at AI Forensics, told Wired that most of the tested Spaces could be used to create nonconsensual intimate images and that users were doing so. He said no platform-wide safeguards were being applied, leaving protections to individual developers, and added that many developers did not add them.
AI Forensics said it was not accusing Hugging Face of creating the models at issue. Bouchaud said the company could filter what enters and exits the systems it hosts.
What safeguards did researchers recommend?
AI Forensics urged Hugging Face to add filtering at the prompt stage and scanning at the output stage for all Spaces that generate images and video. The group said those protections could block sexualized editing requests and stop harmful content from being produced.
The report adds to broader scrutiny of AI tools that can create nonconsensual intimate images, a problem that has spread as image generators and editing models become easier to access. AI Forensics said the current setup leaves too much responsibility with model developers instead of applying baseline protections across the platform.
This story draws on original reporting from The Verge.