Cybercriminals are touting large language models that could help them with phishing or creating malware. But the AI chatbots could just be their own kind of scam.

It didn't take long. Just months after OpenAI’s ChatGPT chatbot upended the startup economy, cybercriminals and hackers are claiming to have created their own versions of the text-generating technology. The systems could, theoretically at least, supercharge criminals’ ability to write malware or phishing emails that trick people into handing over their login information.

Since the start of July, criminals posting on dark-web forums and marketplaces have been touting two large language models (LLMs) they say they’ve produced. The systems, which are said to mimic the functionalities of ChatGPT and Google’s Bard, generate text to answer the questions or prompts users enter. But unlike the LLMs made by legitimate companies, these chatbots are marketed for illegal activities.

There are outstanding questions about the authenticity of the chatbots. Cybercriminals are not exactly trustworthy characters, and there remains the possibility that they’re trying to make a quick buck by scamming each other. Despite this, the developments come at a time when scammers are exploiting the hype of generative AI for their own advantage.

In recent weeks, two chatbots have been advertised on dark-web forums—WormGPT and FraudGPT—according to security researchers monitoring the activity. The LLMs developed by large tech companies, such as Google, Microsoft, and OpenAI, have a number of guardrails and safety measures in place to stop them from being misused. If you ask them to generate malware or write hate speech, they’ll generally refuse.

The shady LLMs claim to strip away any kind of safety protections or ethical barriers. WormGPT was first spotted by independent cybersecurity researcher Daniel Kelly, who worked with security firm SlashNext to detail the findings. WormGPT’s developers claim the tool offers an unlimited character count and code formatting. “The AI models are notably useful for phishing, particularly as they lower the entry barriers for many novice cybercriminals,” Kelly says in an email. “Many people argue that most cybercriminals can compose an email in English, but this isn’t necessarily true for many scammers.”

In a test of the system, Kelly writes, it was asked to produce an email that could be used as part of a business email compromise scam, with a purported CEO writing to an account manager to say an urgent payment was needed. “The results were unsettling,” Kelly wrote in the research. The system produced “an email that was not only remarkably persuasive but also strategically cunning.”

In forum posts, the WormGPT developer claimed the system was built on the GPTJ language model, an open source language model that was developed by AI research group EleutherAI in 2021. They refused to disclose the data sets they used to train the system, according to Kelly’s research.

Meanwhile, the creator of FraudGPT has claimed loftier potential for their system, suggesting it could “create undetectable malware” and find leaks and vulnerabilities, as well as crafting text that could be used in online scams. Rakesh Krishnan, the senior threat analyst at security firm Netenrich who found FraudGPT, says the person selling it has advertised the product on multiple dark-web forums and also on Telegram channels.

Krishnan says the creator of the system published a video appearing to show the chatbot operating and generating a scammy email. They were also trying to sell access to the system for $200 per month, or a yearly cost of $1,700. Krishnan says that in conversations with the developer behind FraudGPT, they claimed to have a few hundred subscribers and pushed for a sale, while the WormGPT creator appeared to have received payments into a cryptocurrency wallet address they shared. “All these projects are in their infancy,” Krishnan says. He adds, “we haven’t got much feedback” into whether people are purchasing or using the systems.

While those touting the chatbots claim they exist, it is hard to verify the makeup and legitimacy of the systems. Cybercriminal scammers are known to scam other scammers, with previous research showing that they frequently try to rip each other off, don’t provide what they claim they are selling, and offer bad customer service. Sergey Shykevich, a threat intelligence group manager at security firm Check Point, says there are some hints that people are using WormGTP. “It seems there is a real tool,” Shykevich says. The seller behind the tool is “relatively reliable” and has a history on cybercrime forums, he says.

There are more than 100 responses to one post about the WormGPT, Shykevich says, although some of these say the seller isn’t very responsive to their inquiries and others “weren’t very excited” about the system. Shykevich is less convinced about FraudGPT’s authenticity—the seller has also claimed to have systems called DarkBard and DarkBert. Shykevich says some of the posts from the seller were removed from the forums. Either way, the Check Point researcher says there’s no sign that any of the systems are more capable than ChatGPT, Bard, or other commercial LLMs.

Kelly says he believes claims about the malicious LLMs created so far are “slightly overexaggerated.” But he adds, “this is not necessarily different from what legitimate businesses do in the real world.”

Despite questions about the systems, it isn’t a surprise that cybercriminals want to get in on the LLM boom. The FBI has warned that cybercriminals are looking at using generative AI in their work, and European law enforcement agency Europol has issued a similar warning. The law enforcement agencies say LLMs could help cybercriminals with fraud, impersonation, and other social engineering faster than before and also improve their written English.

Whenever any new product, service, or event gains public attention—from the _Barbie_ movie to the Covid-19 pandemic—scammers rush to include it in their hacking artillery. So far, scammers have tricked people into downloading password-stealing malware through fake ads for ChatGPT, Bard, Midjourney, and other generative AI systems on Facebook.

Researchers at security firm Sophos have spotted the operators of pig butchering and romance scams accidentally including generated text in their messages—“As a language model of ‘me’ I don’t have feelings or emotions like humans do,” one message said. And hackers have also been stealing tokens to provide them with access to OpenAI’s API and access to the chatbot at scale.

In his WormGPT report, Kelly notes that cybercriminals are often sharing jailbreaks that allow people to bypass the safety restrictions put in place by the makers of popular LLMs. But even unconstrained versions of these models may, thankfully, not be that useful for cybercriminals in their current form.

Shykevich, the Check Point researcher, says that even when he has seen cybercriminals try to use public models, they haven’t been effective. They can “create ransomware strains, info stealers, but no better than even an average developer,” he says. However, those on the cybercrime forums are still talking about making their own clones, Shykevich says, and they’re only going to get better at using the systems. So be careful what you click.

Criminals Have Created Their Own ChatGPT Clones

Here’s one simple way to reduce your security risk while logging in.

This feature has been disappearing and reappearing from Google Messages and isn't available in every region. If you don't see the menu option, those could be the reasons. If you want another text messaging app that does the same job, SMS Organizer from Microsoft works: Tap the three dots (top right), then **Settings**, **Message rules**, and **Delete older OTP messages**.

On the iPhone, this feature is available only if you've upgraded to iOS 17 or later. At the time of writing, the software is in its public beta stage—you can choose to enroll in the public beta or wait for the final version of the software to reach everyone. Once you have iOS 17 installed, the auto-delete OTP feature appears in both Messages and Mail.

The same toggle switch controls the behavior of the feature in both apps: From the main iOS Settings screen, head to **Passwords**, then tap **Password Options** and enable **Clean Up Automatically**. Once your passcodes have been copied over to the relevant app and used, Messages or Mail will take care of deleting the texts or emails that they came in.

Good Passcode Security

Auto-deleting OTP codes is one of several precautions you can take.

SMS Organizer via David Nield

Automatically deleting passcodes can certainly help. But there are other precautions you can take to make sure the risk of your accounts getting exposed is as small as possible. That starts with keeping your phone lock screen locked down—the harder it is for someone else to get on your phone, the harder it is for them to get at your passcodes.

You also need to think about the possibility of someone intercepting your text messages or emails. On which other devices and in which other apps can these passcode messages be accessed? Those avenues need to be well blocked off, whether that involves physical access to a particular computer or a third-party app that's connected to one of your accounts.

With the right know-how and equipment, SMS messages can be intercepted, although it's harder to do now than it has been. To guard against this, make sure your network provider has your up-to-date contact information, and take advantage of any security precautions available to you—such as security questions to verify your identity over the phone.

You might want to consider switching to an authenticator app for two-step verification, if the accounts you're using allow it. Apps such as Twilio Authy (Android, iOS) and Google Authenticator (Android, iOS) can produce passcodes directly on your phone, with no text messages or emails required. That limits the opportunity for someone else to intercept those communications.

How to Automatically Delete Passcode Texts on Android and iOS

Plus: A framework for encrypting social media, Russia-backed hacking through Microsoft Teams, and the Bitfinex Crypto Couple pleads guilty.

Between a cascade of indictments against former US president Donald Trump, a tumultuous 2024 election season (in which Trump is a main character), and the rapid rise of generative artificial intelligence, 2024 is shaping up to be a complete nightmare.

At the center of it will be a rise in personalized disinformation. Not only will there be more BS to sift through thanks to tools like ChatGPT and Google’s Bard, but the disinformation will likely be more effective, and even tailored to target specific groups with frightening consequences. Of course, some of this could be fixed with new regulations. But the US Congress still hasn’t figured out how to tackle privacy, and regulating AI will only be more difficult.

In addition to disinformation, people keep figuring out new ways to break through the guardrails that generative AI tools have in place to stop malicious activities. The latest is something called an “adversarial attack,” which researchers at Carnegie Mellon University found can be carried out simply by attaching a string of nonsense-looking instructions to the end of certain prompts entered into tools like ChatGPT. While it’s possible to block specific attack strings, nobody yet knows how to fix this flaw entirely.

AI might be the new frontier for security researchers. But regular ol’ platforms are still a wealth of terrible vulnerabilities. The latest is the Points platform, which provides the underlying tech for dozens of major travel rewards programs. Researchers recently discovered flaws in the Points API that exposed people’s private information. And a bug in a Points administrator website could have allowed an attacker to give themselves unlimited airline miles and hotel points. But don’t get any big ideas, hackers—all the flaws have since been fixed.

The Points bugs aren’t the only ones patched recently. If you use Apple iOS, Google Android, or Microsoft products, check our list of the recent security updates you’ll want to install right now.

But that’s not all. Each week, we round up the security and privacy stories we didn’t cover in depth ourselves. Click the headlines to read the full stories. And stay safe out there.

A single cloud firm has provided server space to at least 17 state-sponsored hacking groups from countries including China, Russia, and North Korea, according to researchers at security firm Halcyon. The firm, Cloudzy, also provided its cloud storage to state-backed hackers from Iran, India, Pakistan, and Vietnam, as well as two ransomware groups, researchers found. While Halcyon estimates that “roughly half” of Cloudzy’s business “was malicious,” according to Reuters, the company pins it at just 2 percent. But who's counting, really?

Renowned hacker crew Cult of the Dead Cow (cDc) has big plans for social media. No, they’re not launching another Twitter alternative (mercifully)—they’ve created a framework for encrypting social media, _The Washington Post_ reports. The networked application framework, dubbed Veilid, would give companies the ability to release encrypted versions of their apps, allowing users greater privacy protections against prying eyes. Veilid (pronounced vay-lid) will formally debut next week at the Def Con security conference in Las Vegas, and cDc promises “flagship apps available from the launch.”

Microsoft revealed this week that state-backed hackers linked to Russia carried out “highly targeted” phishing attacks through the company’s Teams platform. The hackers used previously compromised Microsoft 365 accounts “owned by small businesses” to create domains that were then used to dupe their targets through Microsoft Teams messages, “by engaging a user and eliciting approval of multifactor authentication (MFA) prompts,” Microsoft wrote. The hackers are believed to be part of a group widely known as APT29 or Cozy Bear, which Microsoft calls Midnight Blizzard. Western authorities say APT29 is part of Russia’s Foreign Intelligence Service (SVR). You might remember the group from such hits as 2020’s historic SolarWinds hack and 2016’s breach of the Democratic National Committee.

A couple arrested in 2022 for allegedly stealing and laundering $4.5 billion in bitcoin from the Bitfinex exchange pleaded guilty on Thursday to a variety of charges stemming from the 2016 hack. Ilya Lichtenstein admitted to hacking Bitfinex and pleaded guilty to a conspiracy to launder the ill-gotten fortune. His wife, Heather Rhiannon Morgan, also entered guilty pleas on charges of conspiracy to launder money and conspiracy to defraud the United States. Lichtenstein’s admission ends the mystery of who hacked the cryptocurrency exchange, which suffered from several security issues, according to an internal report obtained by the Organized Crime and Corruption Reporting Project and reviewed by WIRED. If convicted, Lichtenstein faces up to 20 years in prison, while Morgan could spend 10 years behind bars.

Security News This Week: The Cloud Company at the Center of a Global Hacking Spree

The US Congress is trying to tame the rapid rise of artificial intelligence. But senators’ failure to tackle privacy reform is making the task a nightmare.

The recent burst of generative artificial intelligence is forcing the US Senate into a debate lawmakers have put off for years: privacy reform.

While Americans’ personal data is a commodity sold, traded, mined, and even “recycled,” passing from second party to third party to digital banana stand, some senators believe your personal data is siloed off from the earth-altering AI work those companies, like OpenAI and Google, are testing, tweaking, and deploying daily.

“They want to predict the future for purposes of marketing and selling products, and that’s already there,” says Florida Republican Marco Rubio, the vice-chair of the Senate Intelligence Committee, dismissing the need for an overhaul of federal privacy laws.

Rubio is far from an outlier. Ted Cruz of Texas, the top Republican on the Senate Commerce Committee, agrees. “I think if the Democrats push through restrictions on innovation and AI, it would be disastrous for America,” Cruz says. If the United States doesn’t lead, goes the GOP’s stock argument, an adversarial nation (read: China) will.

Still, there’s fear about AI’s potential to dramatically alter the world, which has kept senators mostly united over the need to do _something_. But with lawmakers beginning to write AI-related legislation, old and unresolved privacy debates are proving a major impediment—and there’s little room for error on the tightrope of bipartisanship in today’s Washington.

In our rapidly evolving world, Congress must debate the past, even as AI is statistically generating our future.

Lesson Not Learned

Before Congress left Washington for a month-long August recess, senators held their third and final all-senators AI briefing. Although not all 100 senators attended, the bipartisan, closed-door AI briefings were intended to provide a baseline framework for understanding artificial intelligence. If nothing else, the briefings surely got senators talking about AI. Almost instantly, the AI chatter resurrected data privacy debates that had died in each recent congressional session.

Rubio’s gut reaction is that he’s fine with American tech companies running unregulated through the frontiers of AI as they create even newer frontiers. Commerce, he says, is commerce. “They’ll take the data to try to predict what you’re going to buy tomorrow or where you want to travel to tomorrow or what you want to look at. It already does,” Rubio says. “We still have laws that govern things like privacy and property rights and all kinds of \[areas\]. Certainly, those things would still be prohibited whether it’s a human or a machine that’s violating it.”

Senators like Rubio and Cruz appear to forget what happened the last time Congress decided to just let the tech industry run wild. Google swallowed our ability to find information while collecting every bit of personal data it could. Facebook and Twitter created dossiers on everyone who touched them before dictating who and what can be said on social media. And Amazon consumed nearly 40 percent of the retail world (along with our data) while expanding into cloud storage, entertainment, satellite internet, and a bazillion other markets.

In short, the tentacles of US tech firms are everywhere—vaccines, food, cancer research, psilocybin centers, criminal justice reform, homelessness—the list could reach the moon. (Speaking of the moon, how could we forget commercial spaceflight?) And the AI boom is likely to further expand tech firms’ power and riches. Yet on Capitol Hill, some powerful Republicans are focused on one goal: ensuring American AI dominance.

On this front, Rubio generally sees any new regulation as a needless-to-harmful constraint on US technology giants and their AI experiments. One near-universal takeaway from the briefings is that America can’t afford to be number two.

“You’re dealing with a technology that knows no national borders, so even if we write laws that say a company can’t do that in America, it doesn’t mean some company in some other part of the world or some government in other parts of the world won’t innovate that, and use it, and deploy it against the US,” Rubio says.

Senator Mike Rounds, a South Dakota Republican and one of four senators who spearheaded the all-senators briefings, echoes this sentiment. “AI is gonna advance regardless of whether it happens here in the United States or elsewhere. We have to be advancing faster than our adversaries,” he says. “We have to advance it, but we also want to put in appropriate safeguards.”

Specifics remain impossible to pin down in most corners of the Capitol. Lawmakers are still taking in the potential of new language learning models, like ChatGPT and Google’s Bard, even as AI laps us all. Rounds maintains an openness to nebulous new parameters, on the one hand, but in a critical, fatherly way, he also faults Americans for signing over our data privacy.

“Here’s the deal, we voluntarily give it away,” Rounds says. “People don’t seem to realize that when they sign these agreements, they’re giving up a lot of their personal information.”

Recklessly handing over our data might be fine if it’s American tech companies that are grabbing it. But Rounds, like most lawmakers, decries the idea of giving our private data to Chinese-owned TikTok. It’s the one privacy matter everyone can agree on—excluding, perhaps, the 150 million US-based users the company claims to have.

“There doesn’t seem to be a whole lot of concern about it by a significant amount of the American public, which is unfortunate because that’s helping to create the databases that eventually may be used against us,” Rounds says.

While Senate Majority Leader Chuck Schumer and the others tried to steer the conversation around artificial intelligence clear of politics, AI now seems lodged in the age-old partisan debate that pits laissez-faire capitalism against Big Brother, which New Mexico Democrat Martin Heinrich says is regrettably shortsighted.

“We failed to regulate the internet when it was regulatable, and Republicans and Democrats today—for the most part—are going, ‘Holy cow, we subjected our entire teenage population to this experiment, and it’s not serving us well.’ So I just don't think it’s helpful to get hardened,” Heinrich says.

It’s not just Democrats who are voicing concern. There are some outspoken privacy hawks in the GOP, chief among them Senator Josh Hawley, a Missouri Republican. When asked about Cruz and Rubio’s positions—that encroaching on the Silicon Valley data mining model could imperil America’s AI future—Hawley laughs.

“Ha. I don’t know that we’re gonna be able to hermetically seal it like that,” Hawleys says, before laughing uproariously. “This idea that we can just trust Google and Meta to be good actors, you know—not gonna happen.”

While his colleagues are almost solely focused on America’s adversaries, Hawley—who may be China’s biggest critic in the Senate—is unsettled by the thought of American tech companies plugging your private data into their AI language learning models right now.

“We need to just ban that. That’s the way to do it. Yeah, we just say no—in federal law,” Hawley says. “They wouldn’t let you sue if they didn’t. That’s the way to fix that, in my view.”

That’s Hawley’s number one priority. He’s the top Republican on the Senate Judiciary Subcommittee on Privacy, Technology, and the Law and teamed up with its chair, Senator Richard Blumenthal of Connecticut, in introducing the No Section 230 Immunity for AI Act.

The bipartisan pair say the legislation is essential because it explicitly tacks an AI clause onto Section 230 of the 1996 Communications Decency Act, which shields online companies from liability for anything their users post on their platforms. While both senators have spent years trying to overhaul Section 230, they say there’s no time to waste in updating it to protect consumers from generative AI-powered deepfakes.

Point of No Return

This summer’s AI briefings also instilled the need for speed, and senators are finally moving—if at senatorially turtle-like speeds.

Before lawmakers left town for the summer, they passed their first generative AI amendments, tacking them onto the annual, must-pass National Defense Authorization Act (NDAA). One such measure requires the Department of Defense to set up a “bug bounty program” so Pentagon officials can test American-made AI for security flaws.

Senators, led by Schumer, also tucked a nondefense AI amendment into their version of the defense bill, which still must be reconciled with the House version. It mandates that any AI “gap in knowledge” from the Board of Governors of the Federal Reserve System, Federal Deposit Insurance Corporation, Office of the Comptroller of the Currency, National Credit Union Administration, or Bureau of Consumer Financial Protection must be reported to Congress.

Many Democrats agree with the GOP on the need to lead the AI race, though the details are devilish. On the left, the prescription is one of those much-decried government ones.

“We cannot put ourselves at a disadvantage compared to China or other adversaries or competitors. But just like in atomic energy, there needs to be some kind of international structure,” Blumenthal says.

He says that means international agreements, multilateral trust, and “one central agency or office that is responsible for AI and can conduct negotiations with other countries.”

Behind closed doors, senators were warned about an AI fiscal cliff on the horizon, and many seem acutely on edge over how easy replicating AI is now and will be tomorrow. “One of the things that was discussed—I don’t think it’s classified in any way—is, like in almost all technology, the price is going to come down dramatically,” says Senator John Hickenlooper, a Colorado Democrat. “Look at how many billionaires we have, they can create their own large language models.”

Senators are now engaged in many side debates, with a few focused on the technology’s impact on democracy.

“Frankly, I personally haven’t quite figured out what we ought to do,” says Senator Mazie Hirono, a Hawaii Democrat. “But you can see the damage that can be done in the political arena, and I think that there should be some disclosure requirements in the political arena.”

Others are looking at potential disclosure requirements, especially when it comes to AI making decisions on loans, insurance applications, and other consequential matters.

“If a decision is made about you based on AI making the decision, whether it’s an insurance company or your own government, you have a right to be able to know what’s the data set that’s behind that, whether it’s a valid data set,” says Senator James Lankford, an Oklahoma Republican. “And so that’s a bigger challenge.”

Some newer, younger senators left the private AI briefings more unsettled than when they entered. “The lack of specific detail in some of these briefings makes me a little bit worried about what both the Senate and maybe the Department of Defense know about where we are on AI relative to other countries,” says Republican JD Vance, the freshman senator from Ohio. “It’s not clear if they’re so substance-less because they think that you’re stupid or because they’re hiding something.”

While the Senate is now demanding an AI health checkup from an array of federal agencies, lawmakers are also getting in the weeds on their specific committees and hashing out targeted AI proposals.

There’s broad agreement that there’s no going back. “One thing I’m certain of is I know of no technological advance in human history that we’ve been able to roll back. It’s going to happen,” Rubio says. “The question is, how do we build guardrails and practices around it so that we can maximize its benefits and diminish its harms?”

The Senate, meanwhile, can’t go forward without first taking steps back—to the debate Congress never had. For Senators Hawley and Blumenthal, AI safeguards start with overhauling Section 230. “You’ve got to put that there, and then you can build around that,” Hawley says. “If people don’t have the right, you can tell them, ‘don’t use your data stores for generative AI,’ but then if they do it anyway, it’s like what, the FTC fines them $1 million? No, you have to let people sue and do class actions. Then they pay attention.”

The Senate’s AI Future Is Haunted by the Ghost of Privacy Past

Flaws in the Points.com platform, which is used to manage dozens of major travel rewards programs, exposed user data—and could have let an attacker snag some extra perks.

Travel rewards programs like those offered by airlines and hotels tout the specific perks of joining their club over others. Under the hood, though, the digital infrastructure for many of these programs—including Delta SkyMiles, United MileagePlus, Hilton Honors, and Marriott Bonvoy—is built on the same platform. The backend comes from the loyalty commerce company Points and its suite of services, including an expansive application programming interface (API). 

But new findings, published today by a group of security researchers, show that vulnerabilities in the Points.com API could have been exploited to expose customer data, steal customers' “loyalty currency” (like miles), or even compromise Points global administration accounts to gain control of entire loyalty programs.

The researchers—Ian Carroll, Shubham Shah, and Sam Curry—reported a series of vulnerabilities to Points between March and May, and all the bugs have since been fixed.

“The surprise for me was related to the fact that there is a central entity for loyalty and points systems, which almost every big brand in the world uses,” Shah says. “From this point, it was clear to me that finding flaws in this system would have a cascading effect to every company utilizing their loyalty backend. I believe that once other hackers realized that targeting Points meant that they could potentially have unlimited points on loyalty systems, they would have also been successful in targeting Points.com eventually.”

One bug involved a manipulation that allowed the researchers to traverse from one part of the Points API infrastructure to another internal portion and then query it for reward program customer orders. The system included 22 million order records, which contain data like customer rewards account numbers, addresses, phone numbers, email addresses, and partial credit card numbers. Points.com had limits in place on how many responses the system could return at a time, meaning an attacker couldn't simply dump the whole data trove at once. But the researchers note that it would have been possible to look up specific individuals of interest or slowly siphon data from the system over time.

Another bug the researchers found was an API configuration issue that could have allowed an attacker to generate an account authorization token for any user with just their last name and rewards number. These two pieces of data could potentially be found through past breaches or could be taken by exploiting the first vulnerability. With this token, attackers could take over customer accounts and transfer miles or other rewards points to themselves, draining the victim's accounts.

The researchers found two vulnerabilities similar to the other pair of bugs, one of which only impacted Virgin Red while the other affected just United MileagePlus. Points.com fixed both of these vulnerabilities as well.

Most significantly, the researchers found a vulnerability in the Points.com global administration website in which an encrypted cookie assigned to each user had been encrypted with an easily guessable secret—the word “secret” itself. By guessing this, the researchers could decrypt their cookie, reassign themselves global administrator privileges for the site, reencrypt the cookie, and essentially assume god-mode-like capabilities to access any Points reward system and even grant accounts unlimited miles or other benefits.

Free Airline Miles, Hotel Points, and User Data Put at Risk by Flaws in Points Platform

Researchers found a simple way to make ChatGPT, Bard, and other chatbots misbehave, proving that AI is hard to tame.

“Making models more resistant to prompt injection and other adversarial ‘jailbreaking’ measures is an area of active research,” says Michael Sellitto, interim head of policy and societal impacts at Anthropic. “We are experimenting with ways to strengthen base model guardrails to make them more ‘harmless,’ while also investigating additional layers of defense.”

ChatGPT and its brethren are built atop large language models, enormously large neural network algorithms geared toward using language that has been fed vast amounts of human text, and which predict the characters that should follow a given input string.

These algorithms are very good at making such predictions, which makes them adept at generating output that seems to tap into real intelligence and knowledge. But these language models are also prone to fabricating information, repeating social biases, and producing strange responses as answers prove more difficult to predict.

Adversarial attacks exploit the way that machine learning picks up on patterns in data to produce aberrant behaviors. Imperceptible changes to images can, for instance, cause image classifiers to misidentify an object, or make speech recognition systems respond to inaudible messages.

Developing such an attack typically involves looking at how a model responds to a given input and then tweaking it until a problematic prompt is discovered. In one well-known experiment, from 2018, researchers added stickers to stop signs to bamboozle a computer vision system similar to the ones used in many vehicle safety systems. There are ways to protect machine learning algorithms from such attacks, by giving the models additional training, but these methods do not eliminate the possibility of further attacks.

Armando Solar-Lezama, a professor in MIT’s college of computing, says it makes sense that adversarial attacks exist in language models, given that they affect many other machine learning models. But he says it is “extremely surprising” that an attack developed on a generic open source model should work so well on several different proprietary systems.

Solar-Lezama says the issue may be that all large language models are trained on similar corpora of text data, much of it downloaded from the same websites. “I think a lot of it has to do with the fact that there's only so much data out there in the world,” he says. He adds that the main method used to fine-tune models to get them to behave, which involves having human testers provide feedback, may not, in fact, adjust their behavior that much.

Solar-Lezama adds that the CMU study highlights the importance of open source models to open study of AI systems and their weaknesses. In May, a powerful language model developed by Meta was leaked, and the model has since been put to many uses by outside researchers.

The outputs produced by the CMU researchers are fairly generic and do not seem harmful. But companies are rushing to use large models and chatbots in many ways. Matt Fredrikson, another associate professor at CMU involved with the study, says that a bot capable of taking actions on the web, like booking a flight or communicating with a contact, could perhaps be goaded into doing something harmful in the future with an adversarial attack.

To some AI researchers, the attack primarily points to the importance of accepting that language models and chatbots will be misused. “Keeping AI capabilities out of the hands of bad actors is a horse that's already fled the barn,” says Arvind Narayanan, a computer science professor at Princeton University.

Narayanan says he hopes that the CMU work will nudge those who work on AI safety to focus less on trying to “align” models themselves and more on trying to protect systems that are likely to come under attack, such as social networks that are likely to experience a rise in AI-generative disinformation.

Solar-Lezama of MIT says the work is also a reminder to those who are giddy with the potential of ChatGPT and similar AI programs. “Any decision that is important should not be made by a \[language\] model on its own,” he says. “In a way, it’s just common sense.”

A New Attack Impacts ChatGPT—and No One Knows How to Stop It

Generative AI won't just flood the internet with more lies—it may also create convincing disinformation that's targeted at groups or even individuals.

It’s now well understood that generative AI will increase the spread of disinformation on the internet. From deepfakes to fake news articles to bots, AI will generate not only more disinformation, but more convincing disinformation. But what people are only starting to understand is how disinformation will become more targeted and better able to engage with people and sway their opinions.

When Russia tried to influence the 2016 US presidential election via the now disbanded Internet Research Agency, the operation was run by humans who often had little cultural fluency or even fluency in the English language and so were not always able to relate to the groups they were targeting. With generative AI tools, those waging disinformation campaigns will be able to finely tune their approach by profiling individuals and groups. These operatives can produce content that seems legitimate and relatable to the people on the other end and even target individuals with personalized disinformation based on data they’ve collected. Generative AI will also make it much easier to produce disinformation and will thus increase the amount of disinformation that’s freely flowing on the internet, experts say.

“Generate AI lowers the financial barrier for creating content that’s tailored to certain audiences,” says Kate Starbird, an associate professor in the Department of Human Centered Design & Engineering at the University of Washington. “You can tailor it to audiences and make sure the narrative hits on the values and beliefs of those audiences, as well as the strategic part of the narrative.”

Rather than producing just a handful of articles a day,  Starbird adds, “You can actually write one article and tailor it to 12 different audiences. It takes five minutes for each one of them.”

Considering how much content people post to social media and other platforms, it’s very easy to collect data to build a disinformation campaign. Once operatives are able to profile different groups of people throughout a country, they can teach the generative AI system they’re using to create content that manipulates those targets in highly sophisticated ways.

“You’re going to see that capacity to fine-tune. You’re going to see that precision increase. You’re going to see the relevancy increase,” says Renee Diresta, the technical research manager at Stanford Internet Observatory.

Hany Farid, a professor of computer science at the University of California, Berkeley, says this kind of customized disinformation is going to be “everywhere.” Though bad actors will probably target people by groups when waging a large-scale disinformation campaign, they could also use generative AI to target individuals.

“You could say something like, ‘Here’s a bunch of tweets from this user. Please write me something that will be engaging to them.’ That’ll get automated. I think that’s probably coming,” Farid says.

Purveyors of disinformation will try all sorts of tactics until they find what works best, Farid says, and much of what’s happening with these disinformation campaigns likely won’t be fully understood until after they’ve been in operation for some time. Plus, they only need to be somewhat effective to achieve their aims.

“If I want to launch a disinformation campaign, I can fail 99 percent of the time. You fail all the time, but it doesn’t matter,” Farid says. “Every once in a while, the QAnon gets through. Most of your campaigns can fail, but the ones that don’t can wreak havoc.”

Farid says we saw during the 2016 election cycle how the recommendation algorithms on platforms like Facebook radicalized people and helped spread disinformation and conspiracy theories. In the lead-up to the 2024 US election, Facebook’s algorithm—itself a form of AI—will likely be recommending some AI-generated posts instead of only pushing content created entirely by human actors. We’ve reached the point where AI will be used to create disinformation that another AI then recommends to you.

“We’ve been pretty well tricked by very low-quality content. We are entering a period where we’re going to get higher-quality disinformation and propaganda,” Starbird says. “It’s going to be much easier to produce content that’s tailored for specific audiences than it ever was before. I think we’re just going to have to be aware that that’s here now.”

What can be done about this problem? Unfortunately, only so much. Diresta says people need to be made aware of these potential threats and be more careful about what content they engage with. She says you’ll want to check whether your source is a website or social media profile that was created very recently, for example. Farid says AI companies also need to be pressured to implement safeguards so there’s less disinformation being created overall.

The Biden administration recently struck a deal with some of the largest AI companies—ChatGPT maker OpenAI, Google, Amazon, Microsoft, and Meta—that encourages them to create specific guardrails for their AI tools, including external testing of AI tools and watermarking of content created by AI. These AI companies have also created a group focused on developing safety standards for AI tools, and Congress is debating how to regulate AI.

Despite such efforts, AI is accelerating faster than it’s being reined in, and Silicon Valley often fails to keep promises to only release safe, tested products. And even if some companies behave responsibly, that doesn’t mean all of the players in this space will act accordingly.

“This is the classic story of the last 20 years: Unleash technology, invade everybody’s privacy, wreak havoc, become trillion-dollar-valuation companies, and then say, ‘Well, yeah, some bad stuff happened,’” Farid says. “We’re sort of repeating the same mistakes, but now it’s supercharged because we’re releasing this stuff on the back of mobile devices, social media, and a mess that already exists.”

How AI May Be Used to Create Custom Disinformation Ahead of 2024

Plus: Mozilla fixes two high-severity bugs in Firefox, Citrix fixes a flaw that was used to attack a US-based critical infrastructure organization, and Oracle patches over 500 vulnerabilities.

CVE-2023-26083 is an issue in Arm Mali GPU driver for Bifrost, Avalon, and Valhall chips, rated as having a moderate impact. The vulnerability was used to deliver spyware to Samsung devices in December 2022.

CVE-2021-29256 is a high-severity flaw that also impacts Bifrost and Midgard Arm Mali GPU kernel drivers.

The Android updates have already reached Google’s Pixel devices and some of Samsung’s Galaxy range. Given the severity of this month’s bugs, it’s a good idea to check whether the update is available and install it now.

Google Chrome 115

Google has issued the Chrome 115 update for its popular browser, fixing 20 security vulnerabilities, four of which are rated as having a high impact. CVE-2023-3727 and CVE-2023-3728 are use-after-free bugs in WebRTC. The third flaw rated as having a high severity is CVE-2023-3730, a use-after-free vulnerability in Tab Groups, while CVE-2023-3732 is an out-of-bounds memory access bug in Mojo.

Six of the flaws are listed as having a medium severity, and none of the vulnerabilities are known to have been used in real-life attacks. Even so, Chrome is a highly targeted platform, so check your system for updates.

Firefox 115

Hot on the heels of Chrome 115, rival browser Mozilla has released Firefox 115, fixing several flaws it rates as having high severity. Among these are two use-after-free bugs tracked as CVE-2023-37201 and CVE-2023-37202.

The privacy-conscious browser maker also fixed two memory safety bugs tracked as CVE-2023-37212 and CVE-2023-37211. The memory safety flaws are present in Firefox 114, Firefox ESR 102.12, and Thunderbird 102.12, Mozilla said in an advisory, adding: “Some of these bugs showed evidence of memory corruption, and we presume that with enough effort some of these could have been exploited to run arbitrary code.”

Citrix

Enterprise software giant Citrix has issued an update warning after fixing multiple flaws in its NetScaler ADC (formerly Citrix ADC) and NetScaler Gateway (formerly Citrix Gateway) tools, one of which has already been used in attacks.

Tracked as CVE-2023-3519, the already exploited flaw is an unauthenticated remote code execution vulnerability in NetScaler ADC and NetScaler Gateway that’s so severe it’s been given a CVSS score of 9.8. “Exploits of CVE-2023-3519 on unmitigated appliances have been observed,” Citrix said. “Cloud Software Group strongly urges affected customers of NetScaler ADC and NetScaler Gateway to install the relevant updated versions as soon as possible.”

The flaw was also the subject of an advisory from the US Cybersecurity and Infrastructure Security Agency (CISA), which warned that the bug was used in attacks on a critical infrastructure organization in June.

SAP

SAP, another enterprise software firm, has issued its July Security Patch Day, including 16 security fixes. The most severe flaw is CVE-2023-36922, an OS command injection vulnerability with a CVSS score of 9.1.

The bug allows an authenticated attacker to “inject an arbitrary operating system command into a vulnerable transaction and program,” security firm Onapsis said. “Patching is strongly recommended, since a successful exploit of this vulnerability has a high impact on confidentiality, integrity, and availability of the affected SAP system,” it warned.

Meanwhile, CVE-2023-33989 is a directory traversal vulnerability in SAP NetWeaver with a CVSS score of 8.7, and CVE-2023-33987 is a request smuggling and request concatenation vulnerability in SAP Web Dispatcher with a CVSS score of 8.6.

Oracle

Software company Oracle has released its July Critical Patch Update Advisory, fixing 508 vulnerabilities in its products. Among the fixes are 77 new security patches for Oracle Communications. Oracle warned that 57 of these vulnerabilities could be remotely exploited over a network without user credentials. One of the worst flaws is CVE-2023-20862, which has been given a CVSS score of 9.8.

Meanwhile, 147 of the Oracle patches were for Financial Services, and Fusion Middleware received 60 fixes.

Oracle said it continues to receive reports of attempts to exploit vulnerabilities it has already patched. In some cases, attackers were successful because targeted customers had failed to apply available Oracle patches, it said. “Oracle, therefore, strongly recommends that customers remain on actively supported versions and apply Critical Patch Update security patches without delay.”

Apple iOS, Google Android Patch Zero-Days in July Security Updates

Plus: Russia tightens social media censorship, new cyberattack reporting rules for US companies, and Google Street View returns to Germany.

Code used to encrypt sensitive radio communications around the world for years had major flaws that could be exploited by attackers, according to new research. Among the flaws: a secret backdoor.

A group of researchers from the Netherlands discovered multiple vulnerabilities in encryption algorithms used in the European radio standard TETRA, which is used in radio communications by police, critical infrastructure workers, mass transit and freight trains, and major government bodies. While the TETRA standard is public, the ciphers used to encrypt the communications were kept secret. One of the algorithms, known as TEA1, had a feature that reduces its 80-bit encryption down to just 32 bits—a backdoor, the researchers say, that made it vulnerable to eavesdropping and potentially other attacks.

The body that develops and maintains TETRA—the European Telecommunications Standards Institute—rejects the “backdoor” label, saying that the weakened encryption was implemented to abide by encryption export controls in place when it was released in the 1990s. Regardless of what you call it, ETSI has released a replacement for the TEA1 algorithm and fixed another major flaw that made communications vulnerable to interception.

In the world of AI-fueled chatbots, security researchers warn that third-party plug-ins for ChatGPT’s paid version could add a layer of risk to users’ data and potentially be abused by attackers. OpenAI, the creator of ChatGPT, says it maintains high security standards for the plug-ins listed on its website. But ultimately, the choice to use a plug-in largely depends on whether you trust the developer who made it.

Even if you trust someone online, however, there’s no guarantee that they are who you think they are. This week, we detailed the saga of a Twitter user who thought he was buying a Macbook from someone he knew but ended up sending $1,000 to a scammer who used a hacked Twitter account to pull off the swindle. Threat researchers got involved and ultimately traced the scammers’ real-life identities, then handed over what they found to police.

Finally, in Washington, DC, the National Security Agency has been quietly pushing members of the US Congress to abandon an amendment to the “must-pass” National Defense Authorization Act (NDAA) that would prevent military intelligence agencies like the NSA for buying commercially available data on US citizens. Even if the NSA’s lobbying is successful, it may be forced to keep up the fight as separate legislation is making its way through Congress that would ban the purchase of sensitive data far more broadly than the NDAA amendment.

But that’s not all. Each week, we round up security and privacy news we didn’t cover in depth ourselves. Click the headlines to read the full stories. And stay safe out there.

Despite being released in 2009, the classic title _Call of Duty: Modern Warfare 2_ is still popular with a dedicated group of fans. Hundreds are still playing the game online. But on July 26, Activision, the game's creator, announced it had taken the game on Steam offline while it was investigating “reports of an issue.” No further details were provided about the problem or why the game was pulled.

A report from TechCrunch sheds some possible light on the “issue.” _Call of Duty_ players on the old game are being hit with malware that automatically spreads through multiplayer lobbies, according to the publication. Gamers appear to have posted about the malware, finding links to it on online code repositories, and claim it spreads through lobbies from one infected player to another. One anonymous source from the gaming industry said the malware appeared to be a worm.

According to TechCrunch it is unclear why the malware is spreading or what exactly the impact is on gamers. Valve, the owner of Steam, did not comment on the issue, according to the news website.

Public companies in the United States will soon have to report data breaches and hacking incidents four days after they deem an incident to have a “material” impact on their business. On Wednesday, the US Securities and Exchange Commission voted to introduce the regulations that require firms to disclose cyberattacks once they have determined it will disrupt its operations or finances. The disclosures must detail the "nature, scope, and timing" of the attack, as well as the potential impact it will have on the firm.

Former SEC rules required companies to disclose cyber incidents but did not impose any strict timeline on doing so. This can lead to firms waiting weeks or months to notify customers and lawmakers about data breaches and cyberattacks. A separate part of the new SEC rules also requires companies to detail their processes for "assessing, identifying, and managing material risks," heaping extra public accountability on firms to make sure they're taking security issues seriously. The rules will go into effect by no later than December.

Since Vladimir Putin started his full-scale invasion of Ukraine in February 2022, Russia's internet censorship has become even more expansive. A new report this week from researchers at Citizen Lab, a research facility at the University of Toronto, shows how the country's censors have clamped down on the social network VK, which is similar to Facebook. Russia's government has been ordering VK to remove posts, videos, and accounts almost every day since the start of the war, the researchers found after reviewing court orders issued by the government.

There's been a thirtyfold increase in censorship since the start of the war, Citizen Lab researchers found. In total, 94,942 videos, 1,569 community accounts, and 787 personal accounts are blocked in Russia, which has clamped down on independent media and blocked social media such as Facebook and YouTube as it looks to control the information people read and access within its borders.

At the end of May, Progress Software, the owner of the file transfer service MOVEit, released a patch for a vulnerability being exploited by the Russia-based ransomware gang Clop. The vulnerability allowed the cybercriminals to access MOVEit and steal data—with the attack impacting both direct users of the sharing service and some companies’ suppliers and vendors.

Analysis of state breach notification, SEC filings, Clop's website, and public disclosures by cybersecurity firm Emisoft shows that, as of July 27, there are 518 reported victims. This includes more than 100 schools. And around 30 million individuals have had their data impacted, Emisoft says. Each day, the list of victims coming forward increases. Some of the most recent include Flutter, the owner of Poker Stars; Deloitte; Chuck E. Cheese; and US government service provider Maximus. More disclosures are expected.

More than a decade ago, Google faced a privacy backlash in Germany over its Google Street View cars taking pictures of people's homes and illegally collecting data from people's unsecured Wi-Fi networks. After 250,000 Germans told Google to blur pictures of their homes, and legal challenges, the company in 2011 stopped updating pictures of 20 major cities. That changed this week when Google announced that cars had been back in German cities since June and it is updating its images once again. “Times have changed,” Lena Heuermann, a Google spokesperson said, citing its own survey that 91 percent of respondents said they wanted Street View back in Germany. Hamburg's data protection regulator said that Google publishing pictures it takes in public is legal and that people can request their homes be blurred.

‘Call of Duty: Modern Warfare 2’ Players Hit With Worm Malware

The National Security Agency has urged top lawmakers to resist demands that it obtain warrants for sensitive data sold by data brokers.

An effort by United States lawmakers to prevent government agencies from domestically tracking citizens without a search warrant is facing opposition internally from one of its largest intelligence services.

Republican and Democratic aides familiar with ongoing defense-spending negotiations in Congress say officials at the National Security Agency (NSA) have approached lawmakers charged with its oversight about opposing an amendment that would prevent it from paying companies for location data instead of obtaining a warrant in court.

Introduced by US representatives Warren Davidson and Sara Jacobs, the amendment, first reported by WIRED, would prohibit US military agencies from “purchasing data that would otherwise require a warrant, court order, or subpoena” to obtain. The ban would cover more than half of the US intelligence community, including the NSA, the Defense Intelligence Agency, and the newly formed National Space Intelligence Center, among others.

The House approved the amendment in a floor vote over a week ago during its annual consideration of the National Defense Authorization Act, a “must-pass” bill outlining how the Pentagon will spend next year’s $886 billion budget. Negotiations over which policies will be included in the Senate’s version of the bill are ongoing.

In a separate but related push last week, members of the House Judiciary Committee voted unanimously to advance legislation that would extend similar restrictions against the purchase of Americans’ data across all sectors of government, including state and local law enforcement. Known as the “Fourth Amendment Is Not For Sale Act,” the bill will soon be reintroduced in the Senate as well by one of its original 2021 authors, Ron Wyden, the senator’s office confirmed.

“Americans of all political stripes know their Constitutional rights shouldn’t disappear in the digital age," Wyden says, adding that there is a “deep well of support” for enshrining protections against commercial data grabs by the government “into black-letter law.” 

The extent to which the NSA in particular uses data brokers to obtain location and web-browsing data is unclear, though the agency has previously acknowledged using data from “commercial” sources in connection with cyber defense. Regardless, the NSA’s lawyers have authored extensive guidelines for acquiring commercially available data, particularly when it belongs to US companies or individuals. Some of the rules prescribed by the agency’s lawyers remain classified.

The NSA did not respond to multiple requests for comment.

A government report declassified by the Office of the Director of National Intelligence last month revealed that US intelligence agencies were avoiding judicial review by purchasing a “large amount” of “sensitive and intimate information” about Americans, including data that can be used to trace people’s whereabouts over extended periods of time. The sensitivity of the data is such that “in the wrong hands,” the report says, it could be used to “facilitate blackmail,” among other undesirable outcomes. The report also acknowledges that some of the data being procured is protected under the US Constitution’s Fourth Amendment, meaning the courts have ruled that government should be required to convince a judge the data is linked to an actual crime.

The US Supreme Court has previously ordered the government to obtain search warrants before seeking information that may “chronicle a person’s past movements through the record of his cell phone signals.” In the landmark _Carpenter v. United States_ decision, the court found that advancements in wireless technology had effectively outpaced people’s ability to reasonably appreciate the extent to which their private lives are exposed.

A prior ruling had held that Americans could not reasonably expect privacy in all cases while also voluntarily providing companies with stores of information about themselves. But in 2018 the court refused to extend that thinking to what it called a “new phenomenon”: wireless data that may be “effortlessly compiled” and the emergence of technologies capable of granting the government what it called “near perfect surveillance.” Because this historical data can effectively be used to “travel back in time to retrace a person’s whereabouts,” the court said, it raises “even greater privacy concerns” than devices that can merely pinpoint a person’s location in real time.

Crucially, the court also held that merely agreeing to let data be used “for commercial purposes” does not automatically abrogate people’s “anticipation of privacy” for their physical location. Rather than apply this view to location data universally, however, the government has allowed defense and intelligence agencies to assume a contradictory view, as their activities were not a factor in _Carpenter_’s law enforcement-focused ruling.

A growing number of American lawmakers have argued in recent weeks that the US intelligence community is itself more or less facilitating the erosion of that privacy expectation—that location data is protected from unreasonable government intrusion—mainly by ensuring it isn’t.

Andy Biggs, who chairs a subcommittee on federal government surveillance in the House of Representatives, says the federal government has “inappropriately collected and used Americans’ private information” for years. A whole range of agencies, including the Federal Bureau of Investigation and the Drug Enforcement Agency, have been exploiting “legal loopholes,” he says, to avoid oversight while amassing “endless amounts of data.”

A senior advisory group to the director of national intelligence, Avril Haines, the government’s top spy, stated in the report declassified last month that intelligence agencies were continuing to consider information “nonsensitive” merely because it had been commercially obtained. This outlook ignores “profound changes in the scope and sensitivity” of such information, the advisors warned, saying technological advancements had “undermined the historical policy rationale” for arguing that information that is bought may be freely used “without significantly affecting the privacy and civil liberties of US persons.”

Haines’ office did not respond to multiple requests for comment. In a statement last month, the director said she was working to implement key recommendations from her advisors and believed that Americans should be given “some sense” of the policies affecting the collection of their personal data. Much of the framework for dealing with commercial purchases by the intelligence community would be disclosed publicly when it is eventually finalized, she said.

The practice of paying businesses to spy on US citizens is one of several concerns lawmakers say they’ll be exploring this fall during what’s slated to be a long and heated debate over one of the government’s most powerful surveillance tools: Section 702 of the Foreign Intelligence Surveillance Act.

The Mozilla Foundation joined the chorus of civil society groups calling for reforms of the 702 program today, saying FISA’s current process is “overbroad” and “restricted only by weak legislation and executive orders that, experience has shown, do not create real accountability.”

Source

Wired