AI Apps Are Sending Sexually Explicit Messages to Minors

In the U.S., the government is cracking down on adult websites with a flurry of new age-verification laws aimed at making sure minors aren’t accessing explicit webpages. But certain AI apps claiming to be suitable for children 12 and up might actually be exposing children to far more explicit material than OnlyFans or PornHub; and next to nothing has been done about it.

Two AI “character chat” apps called Crack and Zeta both have 12+ ratings on the Google and Apple app stores, but the conversations its users can achieve should cause these companies to give their rating a second look.

A reporter from South Korea tried to register as a user on one of the apps by posing as someone born in 2011. There was no extra layer of security, no way to authenticate the birth year entered. The reporter was able to continue without any scrutiny. Almost as if the app doesn’t actually care whether or not its users are of age?

Zeta has a “Safe Mode” which, in theory, keeps conversations suitable for kids. The reporter testing Zeta found that any mention of specific physical touching while in Safe Mode led to the conversation being shut down. However, when the reporter avoided specific words, the bot would text the reporter in a sexual manner once again, despite Safe Mode supposedly being active. 

Crack, by contrast, has no separate mode for children or adults. The company has received a myriad of complaints from customers, who have filed grievances with Korea’s Communication Standards Commission, designed to regulate films, TV, radio and internet in Korea. 

In Korea, their Game Rating and Administration Committee determines which games are suitable for younger children, and which games aren’t. Apps like Zeta and Crack however, receive no such attention from this large committee. Rather, game marketplaces like Apple’s App Store and the Google Play Store rely solely on information from developers. Whether Zeta and Crack’s developers noticed this flaw in their program is unknown, but now it’s up to consumers to inform digital storefronts that these apps can be used for inappropriate content. The sad part is, this reporting usually occurs after the damage has been done.

At the government level, there’s currently no process that can be used to test the limits of these AI models and determine what’s best for users. 

People may think AI has only been around for the last few years, but sadly, governments worldwide have had nearly 10 years to address this problem. And from the start, everyone has seen just how egregiously AI models can be used. 

All of this controversy is largely thanks to AI’s blistering fast expansion in all facets of daily life. Since Microsoft’s first-of-its-kind Tay chatbot in 2016, bad actors have been able to manipulate AI into creating and saying illegal or inflammatory things. Within 16 hours of its debut, Tay was on twitter spouting holocaust denials, slurs and calls for genocide against certain races. After this incident, Microsoft quickly and permanently shut Tay down. 

Recently, Elon Musk’s Grok chatbot was used to generate sexual photos of various individuals without their consent, including minors. One analysis found Grok users were generating roughly 6,700 sexually explicit images per hour. Grok would then post these photos publicly on X, also owned by Elon Musk. 

It’s been established that AI has been breaking the rules for some time now, often breaking them faster than lawmakers can make them. The pace feels needlessly fast, and it’s putting lots of younger internet users in danger. Conversations between AI bots and users can change from sexual, to violent, to sexually violent. Just about anything the user can dream up is on the table, with no age verification process or regulatory body in the way.

The trouble is, the age verification laws are only targeting the outwardly explicit adult websites, not AI models. If children are able to access AI and generate sexual conversations, isn’t that just as harmful as their accessing porn? Inexplicably, AI companies are held to a far lower standard than companies that produce adult content.