Criminals Have Created Their Own ChatGPT Clones

It didn’t take long. Just months after OpenAI’s ChatGPT chatbot upended the startup economy, cybercriminals and hackers are claiming to have created their own versions of the text-generating technology. The systems could, theoretically at least, supercharge criminals’ ability to write malware or phishing emails that trick people into handing over their login information.

Since the start of July, criminals posting on dark-web forums and marketplaces have been touting two large language models (LLMs) they say they’ve produced. The systems, which are said to mimic the functionalities of ChatGPT and Google’s Bard, generate text to answer the questions or prompts users enter. But unlike the LLMs made by legitimate companies, these chatbots are marketed for illegal activities.

There are outstanding questions about the authenticity of the chatbots. Cybercriminals are not exactly trustworthy characters, and there remains the possibility that they’re trying to make a quick buck by scamming each other. Despite this, the developments come at a time when scammers are exploiting the hype of generative AI for their own advantage.

In recent weeks, two chatbots have been advertised on dark-web forums—WormGPT and FraudGPT—according to security researchers monitoring the activity. The LLMs developed by large tech companies, such as Google, Microsoft, and OpenAI, have a number of guardrails and safety measures in place to stop them from being misused. If you ask them to generate malware or write hate speech, they’ll generally refuse.

The shady LLMs claim to strip away any kind of safety protections or ethical barriers. WormGPT was first spotted by independent cybersecurity researcher Daniel Kelly, who worked with security firm SlashNext to detail the findings. WormGPT’s developers claim the tool offers an unlimited character count and code formatting. “The AI models are notably useful for phishing, particularly as they lower the entry barriers for many novice cybercriminals,” Kelly says in an email. “Many people argue that most cybercriminals can compose an email in English, but this isn’t necessarily true for many scammers.”

In a test of the system, Kelly writes, it was asked to produce an email that could be used as part of a business email compromise scam, with a purported CEO writing to an account manager to say an urgent payment was needed. “The results were unsettling,” Kelly wrote in the research. The system produced “an email that was not only remarkably persuasive but also strategically cunning.”

In forum posts, the WormGPT developer claimed the system was built on the GPTJ language model, an open source language model that was developed by AI research group EleutherAI in 2021. They refused to disclose the data sets they used to train the system, according to Kelly’s research.

Meanwhile, the creator of FraudGPT has claimed loftier potential for their system, suggesting it could “create undetectable malware” and find leaks and vulnerabilities, as well as crafting text that could be used in online scams. Rakesh Krishnan, the senior threat analyst at security firm Netenrich who found FraudGPT, says the person selling it has advertised the product on multiple dark-web forums and also on Telegram channels.

Krishnan says the creator of the system published a video appearing to show the chatbot operating and generating a scammy email. They were also trying to sell access to the system for $200 per month, or a yearly cost of $1,700. Krishnan says that in conversations with the developer behind FraudGPT, they claimed to have a few hundred subscribers and pushed for a sale, while the WormGPT creator appeared to have received payments into a cryptocurrency wallet address they shared. “All these projects are in their infancy,” Krishnan says. He adds, “we haven’t got much feedback” into whether people are purchasing or using the systems.

While those touting the chatbots claim they exist, it is hard to verify the makeup and legitimacy of the systems. Cybercriminal scammers are known to scam other scammers, with previous research showing that they frequently try to rip each other off, don’t provide what they claim they are selling, and offer bad customer service. Sergey Shykevich, a threat intelligence group manager at security firm Check Point, says there are some hints that people are using WormGTP. “It seems there is a real tool,” Shykevich says. The seller behind the tool is “relatively reliable” and has a history on cybercrime forums, he says.

There are more than 100 responses to one post about the WormGPT, Shykevich says, although some of these say the seller isn’t very responsive to their inquiries and others “weren’t very excited” about the system. Shykevich is less convinced about FraudGPT’s authenticity—the seller has also claimed to have systems called DarkBard and DarkBert. Shykevich says some of the posts from the seller were removed from the forums. Either way, the Check Point researcher says there’s no sign that any of the systems are more capable than ChatGPT, Bard, or other commercial LLMs.

Kelly says he believes claims about the malicious LLMs created so far are “slightly overexaggerated.” But he adds, “this is not necessarily different from what legitimate businesses do in the real world.”

Despite questions about the systems, it isn’t a surprise that cybercriminals want to get in on the LLM boom. The FBI has warned that cybercriminals are looking at using generative AI in their work, and European law enforcement agency Europol has issued a similar warning. The law enforcement agencies say LLMs could help cybercriminals with fraud, impersonation, and other social engineering faster than before and also improve their written English.

Whenever any new product, service, or event gains public attention—from the Barbie movie to the Covid-19 pandemic—scammers rush to include it in their hacking artillery. So far, scammers have tricked people into downloading password-stealing malware through fake ads for ChatGPT, Bard, Midjourney, and other generative AI systems on Facebook.

Researchers at security firm Sophos have spotted the operators of pig butchering and romance scams accidentally including generated text in their messages—“As a language model of ‘me’ I don’t have feelings or emotions like humans do,” one message said. And hackers have also been stealing tokens to provide them with access to OpenAI’s API and access to the chatbot at scale.

In his WormGPT report, Kelly notes that cybercriminals are often sharing jailbreaks that allow people to bypass the safety restrictions put in place by the makers of popular LLMs. But even unconstrained versions of these models may, thankfully, not be that useful for cybercriminals in their current form.

Shykevich, the Check Point researcher, says that even when he has seen cybercriminals try to use public models, they haven’t been effective. They can “create ransomware strains, info stealers, but no better than even an average developer,” he says. However, those on the cybercrime forums are still talking about making their own clones, Shykevich says, and they’re only going to get better at using the systems. So be careful what you click.