Meta sues maker of explicit deepfake app for dodging its rules to advertise AI ‘nudifying’ tech

Meta is suing the Hong Kong-based maker of the app CrushAI, a platform capable of creating sexually explicit deepfakes, claiming that it repeatedly circumvented the social media company’s rules to purchase ads. The suit is part of what Meta (META) described as a wider effort to crack down on so-called “nudifying” apps — which allow users to create nude or sexualized images from a photo of someone’s face, even without their consent — following claims that the social media giant was failing to adequately address ads for those services on its platforms. As of February, the maker of CrushAI, also known as Crushmate and by several other names, had run more than 87,000 ads on Meta platforms that violated its rules, according to the complaint Meta filed in Hong Kong district court Thursday. Meta alleges the app maker, Joy Timeline HK Limited, violated its rules by creating a network of at least 170 business accounts on Facebook or Instagram to buy the ads. The app maker also allegedly had more than 55 active users managing over 135 Facebook pages where the ads were displayed. The ads primarily targeted users in the United States, Canada, Australia, Germany and the United Kingdom. “Everyone who creates an account on Facebook or uses Facebook must agree to the Meta Terms of Service,” the complaint states. Some of those ads included sexualized or nude images generated by artificial intelligence and were captioned with phrases like “upload a photo to strip for a minute” and “erase any clothes on girls,” according to the lawsuit. CNN has reached out to Joy Timeline HK Limited for comment on the lawsuit. Tech platforms face growing pressure to do more to address non-consensual, explicit deepfakes, as AI makes it easier than ever to create such images. Targets of such deepfakes have included prominent figures such as Taylor Swift and Rep. Alexandria Ocasio-Cortez, as well as high school girls across the United States. The Take It Down Act, which makes it illegal for individuals to share non-consensual, explicit deepfakes online and requires tech platforms to quickly remove them, was signed into law last month. But a series of media reports in recent months suggest that these nudifying AI services have found an audience by advertising on Meta’s platforms. In January, reports from tech newsletter Faked Up and outlet 404Media found that CrushAI had published thousands of ads on Instagram and Facebook and that 90% of the app’s traffic was coming from Meta’s platforms. That’s despite the fact that Meta prohibits ads that contain adult nudity and sexual activity, and forbids sharing non-consensual intimate images and content that promotes sexual exploitation, bullying and harassment. Following those reports, Sen. Dick Durbin, Democrat and ranking member of the Senate Judiciary Committee, wrote to Meta CEO Mark Zuckerberg asking “how Meta allowed this to happen and what Meta is doing to address this dangerous trend.” Earlier this month, CBS News reported that it had identified hundreds of advertisements promoting nudifying apps across Meta’s platforms, including ads that featured sexualized images of celebrities. Other ads on the platforms pointed to websites claiming to animate deepfake images of real people to make them appear to perform sex acts, the report stated. In response to that report, Meta said it had “removed these ads, deleted the Pages responsible for running them and permanently blocked the URLs associated with these apps.” Meta’s efforts to address deepfakes, sexual exploitation Meta says it reviews ads before they run on its platforms, but its complaint indicates that it has struggled to enforce its rules. According to the complaint, some of the CrushAI ads blatantly advertised its nudifying capabilities with captions such as “Ever wish you could erase someone’s clothes? Introducing our revolutionary technology” and “Amazing! This software can erase any clothes.” Now, Meta said its lawsuit against the CrushAI maker aims to prevent it from further circumventing its rules to place ads on its platforms. Meta alleges it has lost $289,000 because of the costs of the investigation, responding to regulators and enforcing its rules against the app maker. When it announced the lawsuit Thursday, the company also said it had developed new technology to identify these types of ads, even if the ads themselves didn’t contain nudity. Meta’s “specialist teams” partnered with external experts to train its automated content moderation systems to detect the terms, phrases and emojis often present in such ads. “This is an adversarial space in which the people behind it — who are primarily financially motivated — continue to evolve their tactics to avoid detection,” the company said in a statement. “Some use benign imagery in their ads to avoid being caught by our nudity detection technology, while others quickly create new domain names to replace the websites we block.” Meta said it had begun sharing information about nudifying apps attempting to advertise on its sites with other tech platforms through a program called Lantern, run by industry group the Tech Coalition. Tech giants created Lantern in 2023 to share data that could help them fight child sexual exploitation online. The push to crack down on deepfake apps comes after Meta dialed back some of its automated content removal systems — prompting some backlash from online safety experts. Zuckerberg announced earlier this year that those systems would be focused on checking only for illegal and “high-severity” violations such as those related to terrorism, child sexual exploitation, drugs, fraud and scams. Other concerns must be reported by users before the company evaluates them.