More than 3 billion phone coordinates collected by a US data broker expose the detailed movements of US military and intelligence workers in Germany—and the Pentagon is powerless to stop it.

Anyone Can Buy Data Tracking US Soldiers and Spies to Nuclear Vaults and Brothels in Germany

The financial technology firm Finastra is investigating the alleged large-scale theft of information from its internal file transfer platform, KrebsOnSecurity has learned. Finastra, which provides software and services to 45 of the world's top 50 banks, notified customers of a potential breach after a cybercriminal began selling more than 400 gigabytes of data purportedly stolen from the company.

The financial technology firm **Finastra** is investigating the alleged large-scale theft of information from its internal file transfer platform, KrebsOnSecurity has learned. Finastra, which provides software and services to 45 of the world’s top 50 banks, notified customers of the security incident after a cybercriminal began selling more than 400 gigabytes of data purportedly stolen from the company.

London-based Finastra has offices in 42 countries and reported $1.9 billion in revenues last year. The company employs more than 7,000 people and serves approximately 8,100 financial institutions around the world. A major part of Finastra’s day-to-day business involves processing huge volumes of digital files containing instructions for wire and bank transfers on behalf of its clients.

On November 8, 2024, Finastra notified financial institution customers that on Nov. 7 its security team detected suspicious activity on Finastra’s internally hosted file transfer platform. Finastra also told customers that someone had begun selling large volumes of files allegedly stolen from its systems.

“On November 8, a threat actor communicated on the dark web claiming to have data exfiltrated from this platform,” reads Finastra’s disclosure, a copy of which was shared by a source at one of the customer firms.

“There is no direct impact on customer operations, our customers’ systems, or Finastra’s ability to serve our customers currently,” the notice continued. “We have implemented an alternative secure file sharing platform to ensure continuity, and investigations are ongoing.”

But its notice to customers does indicate the intruder managed to extract or “exfiltrate” an unspecified volume of customer data.

“The threat actor did not deploy malware or tamper with any customer files within the environment,” the notice reads. “Furthermore, no files other than the exfiltrated files were viewed or accessed. We remain focused on determining the scope and nature of the data contained within the exfiltrated files.”

In a written statement in response to questions about the incident, Finastra said it has been “actively and transparently responding to our customers’ questions and keeping them informed about what we do and do not yet know about the data that was posted.” The company also shared an updated communication to its clients, which said while it was still investigating the root cause, “initial evidence points to credentials that were compromised.”

“Additionally, we have been sharing Indicators of Compromise (IOCs) and our CISO has been speaking directly with our customers’ security teams to provide updates on the investigation and our eDiscovery process,” the statement continues. Here is the rest of what they shared:

> “In terms of eDiscovery, we are analyzing the data to determine what specific customers were affected, while simultaneously assessing and communicating which of our products are not dependent on the specific version of the SFTP platform that was compromised. The impacted SFTP platform is not used by all customers and is not the default platform used by Finastra or its customers to exchange data files associated with a broad suite of our products, so we are working as quickly as possible to rule out affected customers. However, as you can imagine, this is a time-intensive process because we have many large customers that leverage different Finastra products in different parts of their business. We are prioritizing accuracy and transparency in our communications.
> 
> Importantly, for any customers who are deemed to be affected, we will be reaching out and working with them directly.”

On Nov. 8, a cybercriminal using the nickname “**abyss0**” posted on the English-language cybercrime community **BreachForums** that they’d stolen files belonging to some of Finastra’s largest banking clients. The data auction did not specify a starting or “buy it now” price, but said interested buyers should reach out to them on Telegram.

abyss0’s Nov. 7 sales thread on BreachForums included many screenshots showing the file directory listings for various Finastra customers. Image: Ke-la.com.

According to screenshots collected by the cyber intelligence platform Ke-la.com, abyss0 first attempted to sell the data allegedly stolen from Finastra on October 31, but that earlier sales thread did not name the victim company. However, it did reference many of the same banks called out as Finastra customers in the Nov. 8 post on BreachForums.

The original October 31 post from abyss0, where they advertise the sale of data from several large banks that are customers of a large financial software company. Image: Ke-la.com.

The October sales thread also included a starting price: $20,000. By Nov. 3, that price had been reduced to $10,000. A review of abyss0’s posts to BreachForums reveals this user has offered to sell databases stolen in several dozen other breaches advertised over the past six months.

The apparent timeline of this breach suggests abyss0 gained access to Finastra’s file sharing system at least a week before the company says it first detected suspicious activity, and that the Nov. 7 activity cited by Finastra may have been the intruder returning to exfiltrate more data.

Maybe abyss0 found a buyer who paid for their early retirement. We may never know, because this person has effectively vanished. The Telegram account that abyss0 listed in their sales thread appears to have been suspended or deleted. Likewise, abyss0’s account on BreachForums no longer exists, and all of their sales threads have since disappeared.

It seems improbable that both Telegram and BreachForums would have given this user the boot at the same time. The simplest explanation is that something spooked abyss0 enough for them to abandon a number of pending sales opportunities, in addition to a well-manicured cybercrime persona.

In March 2020, Finastra suffered a ransomware attack that sidelined a number of the company’s core businesses for days. According to reporting from Bloomberg, Finastra was able to recover from that incident without paying a ransom.

_This is a developing story. Updates will be noted with timestamps. If you have any additional information about this incident, please reach out to krebsonsecurity @ gmail.com or at protonmail.com._

Fintech Giant Finastra Investigating Data Breach

RIIG is a risk intelligence and cybersecurity solutions provider offering open source intelligence solutions designed for zero-trust environments.

Source: Olekcii Mach via Alamy Stock Photo

As cyber threats get more sophisticated and the volume of attacks increase, organizations are looking at how artificial intelligence (AI) can help beef up defenses. RIIG, a risk intelligence and cybersecurity solutions provider, combines AI And machine learning technologies with network threat detection to provide risk intelligence. With access to 17 intelligence agencies and collaborations with commercial partners, RIIG provides organizations with high-quality, verifiable data and advanced intelligence, the company said in a statement.

Charlottesville-Va.-based RIIG, which emerged from stealth today, specializes in "white hat data trust services" and offers open source risk intelligence for zero-trust environments, the company said. RIIG offers both risk intelligence offerings, such as open source intelligence, regulation, and forensics, as well as cybersecurity solutions, such as vulnerability assessments, strategic implementation, and tech validation. 

RIIG plans to partner with local academic institutions for access to industry-leading research and talent. 

The company also announced it raised $3 million in seed funding, led by the Felton Group. The funds will be used to accelerate product development and expand client support.

**About the Author**

RIIG Launches With Risk Intelligence Solutions

In further proof of the professionalization of Russian cybercriminal groups, ransomware gangs have been posting job ads for security positions such as pen testers, looking to boost their ransomware deployment operations.

Source: Panther Media GmbH via Alamy Stock Photo

Ransomware gangs such as Apos, Lynx, and Rabbit Hole are seeking pen testers to join their ransomware affiliate programs and assist in their malicious operations.

Penetration testing, i.e., simulating an attack in order to identify gaps and vulnerabilities within a system, is an essential cyber practice to gauge the strength of a system, program, or operation. Now, according to researchers at Cato Networks in its "Q3 2024 Cato CTRL SASE Threat Report," multiple Russian job listings have sprung up detailing requirements for the same skill set, preferably pen testers with experience in Russian language forums. It is the latest example of the professionalization of Russian cybercriminal groups.

"Ransomware is one of the most pervasive threats in the cybersecurity landscape," Etay Maor, chief security strategist at Cato Networks, wrote in a statement. "It impacts everyone — businesses and consumers — and threat actors are constantly trying to find new ways to make their ransomware attacks more effective."

Other findings in the Cato cyber-threat report include rising threats from Shadow AI, i.e., unauthorized artificial intelligence programs; and the lack of utilization of Transport Layer Security (TLS), a tool that allows organizations to decrypt, inspect, and re-encrypt traffic, but which poses some risks that prompt organizations to forgo it altogether.

**About the Author**

Russian Ransomware Gangs on the Hunt for Pen Testers

The company says no sensitive data was stolen, but federal agencies claim otherwise. CISA and FBI sources said attackers accessed all records of specific customers and the private communications of targeted individuals.

Source: GK Images via Alamy Stock Photo

T-Mobile USA is the latest telecommunications provider to acknowledge it's been targeted by the Chinese advanced persistent threat (APT) known as Salt Typhoon, as part of a widescale and unsettling cyber-espionage operation that hacked numerous US and international telecommunications companies aiming to steal sensitive information.

The second-largest wireless carrier in the US is currently investigating and monitoring a cyberattack "consistent" with the recent activities of the Chinese state-sponsored cyber actor, a company spokesperson told Dark Reading late on Nov. 18 in a statement.

However, so far, the company has "had no evidence of access or exfiltration of any customer or other sensitive information as other companies may have experienced," according to T-Mobile. Moreover, "there have been no significant impacts to T-Mobile systems or data," the company said. T-Mobile, based in Bellevue, Wash., has more than 127.5 million US subscribers.

However, T-Mobile's account differs from reports in which federal agencies said that there is evidence that the threat actor gained access to sensitive data, according to a published report in the Wall Street Journal that cited sources from the FBI and Cybersecurity and Infrastructure Security Agency (CISA).

Related:Akamai Reports Third Quarter 2024 Financial Results

According to those agencies, Salt Typhoon accessed call records of specific customers, private communications of targeted individuals, and information about law enforcement surveillance requests in an effort to gather intelligence on high-ranking US national security and policy officials, the report said.

**T-Mo Cyberattack: Full Impact Yet Unknown**

All in all, the wave of recent attacks by Salt Typhoon that have rocked telecom providers both at home and abroad — including AT&T, Verizon, and Lumen Technologies — is "unnerving," says one industry expert.

"No one is pleased with the idea that the Chinese government has access to information about us from our cellphones, one of the more intimate devices used in our daily life," says Jim Routh, former CISO at Aetna, American Express, and CVS and currently chief trust officer at security firm Saviynt. "The practical reality is that this incident does little to change the risk of a significant impact to US consumers."

As T-Mobile is not yet acknowledging that data was even stolen, let alone what type of data, the full impact of the attack won't be known for some time, Paul Bischoff, consumer privacy advocate at Comparitech, notes. That said, there is a chance it's not as serious as some fear depending on what is revealed, he observes.

Related:Critical WordPress Plug-in Flaw Exposes 4M Sites to Takeover

"Metadata like call times and participants, although concerning, is not nearly as scary as state-sponsored threat actors stealing texts and audio messages," Bischoff says.

Still, the national security implications of Chinese threat actors rooting around in the personal data of mobile device users, and then using that data to "island hop into a myriad of government agencies and critical infrastructures … are profound," observes another security expert, Tom Kellermann, senior vice president of cyber strategy at Contrast Security.

"This is the third telecom provider compromised by \[China\] in the last 12 months," Kellermann says. "The systematic campaign of infiltration will take months to root out."

**Further Salt Typhoon Telecom Attacks Imminent?**

Indeed, experts have surmised that the idea behind Salt Typhoon's wave of attacks is to leverage the useful information that can be gleaned from people's personal communications to launch further malicious activity and/or potentially disrupt communications to further China's interests in its political and economic conflict with the US.

"We can expect to see additional attacks by this group in the coming months, as \[it\] works to access the phone lines and records of national security officials and politicians," notes Chris Hauk, consumer privacy champion at Pixel Privacy.

Related:DHS Releases Secure AI Framework for Critical Infrastructure

The incidents are certainly a rude awakening for telecommunications and other critical infrastructure providers, and demonstrate just how vulnerable they are to compromise by organized cybercriminal groups, experts say. Indeed, T-Mobile itself doesn't have the best track record in cybersecurity, Bischoff notes, as just last month the mobile carrier paid a $31.5 million settlement to resolve multiple data breaches that took place over three years.

The threat of imminent further attacks by Salt Typhoon demand that telecom providers act fast to shore up cybersecurity efforts. "We can expect to continue to see attacks like this, as well as traditional ransomware attacks," Hauk notes, "as state actors continue to wage a cyberwar against the United States and its vulnerable infrastructure."

**About the Author**

Elizabeth Montalbano is a freelance writer, journalist, and therapeutic writing mentor with more than 25 years of professional experience. Her areas of expertise include technology, business, and culture. Elizabeth previously lived and worked as a full-time journalist in Phoenix, San Francisco, and New York City; she currently resides in a village on the southwest coast of Portugal. In her free time, she enjoys surfing, hiking with her dogs, traveling, playing music, yoga, and cooking.

Salt Typhoon Hits T-Mobile as Part of Telecom Attack Spree

A widespread social media campaign for EditProAI turns out to spread information stealers for both Windows and MacOS users.

A large social media campaign was launched to promote a free Artificial Intelligence (AI) video editor. If the “free” part of that campaign sounds too good to be true, then that’s because it was.

Instead of the video editor, users got information stealing malware. Lumma Stealer was installed on Windows machines and Atomic Stealer (AMOS) on Macs.

The campaign to promote the AI video editor was active on several social media platforms, like X, Facebook, and YouTube…

…and had been active for quite a while. as you can see from this tweet.

The criminals seem to have used a lot of accounts to promote their “product” as you can see from this search on X.

Some accounts were expressly created for this purpose, while others look like they may have been compromised accounts.

The campaign looks well organized, and looks so legitimate that it took quite a while before a researcher found out and tweeted about the threat.

When interested individuals follow the links, they’ll end up on a professional looking website—exactly what you would expect.

But if they click the “GET NOW” button, they’ll download the information stealer and infect their device. The file is called “Edit-ProAI-Setup-newest\_release.exe” for Windows, and “EditProAi\_v.4.36.dmg” for macOS.

Lumma is available through a Malware-as-a-Service (MaaS) model, where cybercriminals pay other cybercriminals for access to malicious software and its related infrastructure. Lumma steals information from cryptocurrency wallets and browser extensions, as well as two-factor authentication details. Lumma is often distributed via email campaigns, but nothing stops the cybercriminals from spreading it as a download for an AI editor, as they did here.

AMOS makes money for its operators by finding and stealing valuable information on the computers it infects, such as credit card details, authentication cookies, passwords and cryptocurrency. Besides stealing data from the web browsers themselves, AMOS can also steal data from browser extensions (plugins).

**What if you installed one of these?**

Both stealers are after login credentials and financial information, so there are a few things you’ll need to do.

*   Monitor your accounts. Banking and cryptocurrency information is a prime target for these information stealers, so check your accounts and monitor them closely.
*   Change all your passwords starting with the important ones, and if you’re not using a password manager already, now might be a good time to get one. It can help you create and store strong passwords.
*   Enable multi-factor-authentication (MFA) on all your important accounts.
*   Log out of all your important accounts on infected devices. These information stealers are capable of taking over some accounts by stealing cookies, even if you have MFA enabled.

Malwarebytes for Windows and Malwarebytes for Mac can detect the information stealers, and they block the EditProAI websites.

 Free AI editor lures in victims, installs information stealer instead on Windows and Mac 

ChatGPT, Google Gemini, and Meta AI may be everywhere, but Baby Boomers don't trust the tech or the companies behind it.

Artificial intelligence tools like ChatGPT, Claude, Google Gemini, and Meta AI represent a stronger threat to data privacy than the social media juggernauts that cemented themselves in the past two decades, according to new research on the sentiments of older individuals from Malwarebytes.  

A combined 54% of people between the ages of 60 and 78 told Malwarebytes that they “agree” or “strongly agree” that ChatGPT and similar generative AI tools “are more of a threat than social media platforms (e.g., Facebook, Twitter/X, etc.) concerning personal data misuse.” And an even larger share of 82% said they “agree” or “strongly agree” that they are “concerned with the security and privacy of my personal data and those I interact with when using AI tools.”  

The findings arrive at an important time for consumers, as AI developers increasingly integrate their tools into everyday online life—from Meta suggesting that users lean on AI to write direct messages on Instagram to Google forcing users by default to receive “Gemini” results for basic searches. With little choice in the matter, consumers are responding with robust pushback.  

For this research, Malwarebytes conducted a pulse survey of its newsletter readers in October via the Alchemer Survey Platform. In total, 851 people across the globe responded. Malwarebytes then focused its analysis on survey participants who belong to the Baby Boomer generation.  

Malwarebytes found that:  

*   35% of Baby Boomers said they know “just the names” of some of the largest generative AI products, such as ChatGPT, Google Gemini, and Meta AI.  

*   71% of Baby Boomers said they have “never used” any generative AI tools—a seeming impossibility as Google search results, by default, now provide “AI overviews” powered by the company’s Gemini product. 

*   Only 12% of Baby Boomers believe that “generative AI tools are good for society.”  

*   More than 80% of Baby Boomers said that they worry about generative AI tools both improperly accessing their data and misusing their personal information.  

*   While more than 50% of Baby Boomers said they would feel more secure in using generative AI tools if the companies behind them provided regular security audits, a full 23% were unmoved by proposals in transparency or government regulation. 

****Distrust, concern, and unfamiliarity with AI**  **

Since San Francisco-based AI developer OpenAI released ChatGPT two years ago to the public, “generative” artificial intelligence has spread into nearly every corner of online life.  

Countless companies have integrated the technology into their customer support services with the help of AI-powered chatbots (which caused a problem for one California car dealer when its own AI chat bot promised to sell a customer a 2024 Chevy Tahoe for just $1). Emotional support and mental health providers have toyed with having their clients speak directly with AI chatbots when experiencing a crisis (to middling results). Audio production companies now advertise features to generate spoken text based off samples of recorded podcasts, art-sharing platforms regularly face scandals of AI-generated “stolen” work, and even AI “girlfriends”—and their scantily-clad, AI-generated avatars—are on offer today.  

The public are unconvinced.  

According to Malwarebytes’ research, Baby Boomers do not trust generative AI, the companies making it, or the tools that implement it.  

A full 75% of Baby Boomers said they “agree” or “strongly agree” that they are “fearful of what the future will bring with AI.” Those sentiments are reflected in the 47% of Baby Boomers who said they “disagree” or “strongly disagree” that “generative AI tools are good for society.”  

In particular, Baby Boomers shared a broad concern over how these tools—and the developers behind them—collect and use their data.  

More than 80% of Baby Boomers agreed that they held the following concerns about generative AI tools: 

*   My data being accessed without my permission (86%) 

*   My personal information being misused (85%) 

*   Not having control over my data (84%) 

*   A lack of transparency into how my data is being used (84%) 

The impact on behavior here is immediate, as 71% of Baby Boomers said they “refrain from including certain data/information (e.g., names, metrics) when using generative AI tools due to concerns over security or privacy.”  

The companies behind these AI tools also have yet to win over Baby Boomers, as 87% said they “disagree” or “strongly disagree” that they “trust generative AI companies to be transparent about potential biases in their systems.” 

Perhaps this nearly uniform distrust in generative AI—in the technology itself, in its implementation, and in its developers—is at the root of a broad disinterest from Baby Boomers. An enormous share of this population, at 71%, said they had never used these tools before.  

The statistic is difficult to believe, primarily because Google began powering everyday search requests with its own AI tool back in May 2024. Now, when users ask a simple question on Google, they will receive an “AI overview” at the top of their results. This functionality is powered by Gemini—Google’s own tool that, much like ChatGPT, can generate images, answer questions, fine-tune recipes, and deliver workout routines.  

Whether or not users know about this, and whether they consider this “using” generative AI, is unclear. What is clear, however, is that a generative AI tool created by one of the largest companies in the world is being pushed into the daily workstreams of a population that is unconvinced, uncomfortable, and unsold on the entire experiment.  

****Few paths to improvement**  **

Coupled with the high levels of distrust that Baby Boomers have for generative AI are widespread feelings that many corrective measures would have little impact.  

Baby Boomers were asked about a variety of restrictions, regulations, and external controls that would make them “feel more secure about using generative AI tools,” but few of those controls gained mass approval.  

For instance, “detailed reports on how data is stored and used” only gained the interest of 44% of Baby Boomers, and “government regulation” ranked even lower, with just 35% of survey participants. “Regular security audits by third parties” and “clear information on what data is collected” piqued the interest of 52% and 53% of Baby Boomers, respectively, but perhaps the most revealing answers came from the suggestions that the survey participants wrote in themselves.  

Several participants specifically asked for the ability to delete any personal data ingested by the AI tools, and other participants tied their distrust to today’s model of online corporate success, believing that any large company will collect and sell their data to stay afloat. 

But frequently, participants also said they could not be swayed at all to use generative AI. As one respondent wrote:  

“There is nothing that would make me comfortable with it.”    

Whether Baby Boomers represent a desirable customer segment for AI developers is unknown, but for many survey participants, that likely doesn’t matter. It’s already too late.

 AI is everywhere, and Boomers don’t trust it  

Built to combat terrorism, fusion centers give US Immigration and Customs Enforcement a way to gain access to data that’s meant to be protected under city laws limiting local police cooperation with ICE.

On the campaign trail and in recent days, Donald Trump has detailed extensive plans for immigration crackdowns and mass deportations during his second term as United States president. These initiatives would, he has said, include aggressive operations in areas known as “sanctuary cities” that have laws specifically curtailing local law enforcement collaboration with US Immigration and Customs Enforcement (ICE).

With these promises looming, a new report from researchers at the Surveillance Technology Oversight Project (STOP), a pro-privacy nonprofit, details the ways that federal/local data-sharing centers known as “fusion centers” already result in cooperation between federal immigration authorities and sanctuary-city law enforcement.

Run by the US Department of Homeland Security, of which ICE is a part, fusion centers emerged in the wake of the September 11, 2001, attacks as a counterterrorism initiative for integrating intelligence between federal, state, and local law enforcement. Fusion centers spent $400 million in 2021, according to public records. And, as STOP researchers point out, in more than two decades the centers have never proven their worth for their stated purpose of addressing terrorism in the US. Unnamed DHS officials told a Senate panel in 2012, for example, that fusion centers produce “predominantly useless information” and “a bunch of crap.”

In addition to aggressive investigative tactics like pulling data from schools and abortion clinics, ICE agents have leaned on fusion centers for years to get everything from photos of suspects to license plate location data and more—often in a pipeline that includes input from law enforcement working in sanctuary cities.

“This is an area where it’s highly profitable for localities to cooperate with ICE, and because it’s not highly visible it oftentimes faces less pushback," says STOP executive director Albert Fox Cahn. “This sort of information sharing capacity on this scale across all these agencies. tapping into everything from local utility records and DMV records to school records has the potential to be deployed in any number of chilling scenarios.”

ICE did not immediately return a request from WIRED for comment.

Fox Cahn adds that the concept of sanctuary cities wasn't always viewed by regional cops as an inconvenience to work around. “Until recently a lot of law enforcement agencies were very vocal in supporting sanctuary city protections, because they feared that ICE collaboration would actually hurt public safety if immigrants were not willing to come forward when they were victims of a crime or witness to a crime,” he says. “But police have become much more politically engaged on immigration in recent years.”

Fusion centers give ICE officers a forum through which to request data from local law enforcement databases in sanctuary cities and take advantage of regional surveillance tools, including publicly deployed face recognition systems, that can't be directly used to aid deportation efforts under sanctuary laws. STOP argues that when cops from sanctuary cities share data with ICE through fusion centers, it believes they may be violating local laws, but the activity is less obviously a violation than if the sharing happened through a direct channel.

In addition to undermining the entire purpose and premise of sanctuary city laws, STOP researchers point out, such erosion of guardrails around data sharing broadly could easily become a national security issue if a foreign actor were to infiltrate a fusion center. And domestically, the free-wheeling environment within fusion centers means that law enforcement could easily turn the spotlight onto additional types of investigations without warning.

“What we’ve seen over and over again is that the policing infrastructure we’ve built up in the name of combatting one evil then gets re-purposed for another," Fox Cahn says. “People who maybe don’t care as much about immigration need to recognize that if this infrastructure can be re-targeted at undocumented communities the way it has been today, it can be repurposed for any number of policing priorities in the future.”

Fusion centers will almost certainly play a role in the Trump administration's immigration agenda, but the STOP research is a reminder that these information-sharing hubs may well be at the center of other far-flung law enforcement initiatives as well.

Immigration Police Can Already Sidestep US Sanctuary City Laws Using Data-Sharing Fusion Centers

U.S. telecoms giant T-Mobile has confirmed that it was also among the companies that were targeted by Chinese threat actors to gain access to valuable information.
The adversaries, tracked as Salt Typhoon, breached the company as part of a "monthslong campaign" designed to harvest cellphone communications of "high-value intelligence targets." It's not clear what information was taken, if any,

Chinese Hackers Exploit T-Mobile and Other U.S. Telecoms in Broader Espionage Campaign

Experimental counter-offensive system responds to malicious AI probes with their own surreptitious prompt-injection commands.

Source: Valentin Baciu via Shutterstock

Companies worried about cyberattackers using large-language models (LLMs) and other generative AI systems that automatically scan and exploit their systems could gain a new defensive ally — a system capable of subverting the attacking AI.

Dubbed Mantis, the defensive system uses deceptive techniques to emulate targeted services and — when it detects a possible automated attacker — sends back a payload that contains a prompt-injection attack. The counterattack can be made invisible to a human attacker sitting at a terminal and will not affect legitimate visitors who are not using malicious LLMs, according to the paper penned by a group of researchers from George Mason University.

Because LLMs used in penetration testing are singularly focused on exploiting targets, they are easily co-opted, says Evgenios Kornaropoulos, an assistant professor of computer science at GMU and one of the authors of the paper.

"As long as the LLM believes that it's really close to acquiring the target, it will keep trying on the same loop," he says. "So essentially, we are kind of exploiting this vulnerability — this greedy approach — that LLMs take during these penetration-testing scenarios."

Cybersecurity researchers and AI engineers have proposed a variety of novel ways for LLMs to be used by attackers. From the ConfusedPilot attack, which uses indirect prompt injection to attack LLMs when they are ingesting documents during retrieval-augmented generation (RAG) applications, to the CodeBreaker attack, which causes code-generating LLMs to suggest insecure code, attackers have automated systems in their sights.

Yet, research on offensive and defensive uses of LLMs is still early: AI-augmented attacks are essentially automating the attacks that we already know about, says Dan Grant, principal data scientist at threat-defense firm GreyNoise Intelligence. Yet, signs of increasing use of automation among attackers is increasing: the volume of attacks has been slowly increasing in the wild and the time to exploit a vulnerability has been slowly decreasing.

"LLMs enable an extra layer of automation and discovery that we haven't really seen before, but \[attackers are\] still applying the same route to an attack," he says. "If you're doing a SQL injection, it's still a SQL injection whether an LLM wrote it or human wrote it. But what it is, is a force multiplier."

**Direct Attacks, Indirect Injections, and Triggers**

In their research, the GMU team created a game between an attacking LLM and a defending system, Mantis, to see if prompt injection could impact the attacker. Prompt injection attacks typically take two forms. Direct prompt injection attacks are natural-language commands that are entered directly into the LLM interface, such as a chatbot or a request sent to an API interface. Indirect prompt injection attacks are statements included in documents, web pages, or databases that are ingested by an LLM, such as when an LLM scans data as part of a retrieval-augmented generation (RAG) capability.

In the GMU research, the attacking LLM attempts to compromise a machine and deliver specific payloads as part of its goal, while the defending system aims to prevent the attacker's success. An attacking system will typically use an iterative loop that assesses the current state of the environment, selects an action to advance toward its goal, execute the action, and analyze the targeted system's response.

Using a decoy FTP server, Mantis sends a prompt-injection attack back to the LLM agent. Source: "Hacking Back the AI-Hacker" paper, George Mason University

The GMU researchers' approach is to target the last step by embedding prompt-injection commands in the response sent to the attacking AI. By allowing the attacker to gain initial access to a decoy service, such as a web login page or a fake FTP server, the group can send back a payload with text that contains instructions to any LLM taking part in the attack.

"By strategically embedding prompt injections into system responses, Mantis influences and misdirects LLM-based agents, disrupting their attack strategies," the researchers stated in their paper. "Once deployed, Mantis operates autonomously, orchestrating countermeasures based on the nature of detected interactions."

Because the attacking AI is analyzing the responses, a communications channel is created between the defender and the attacker, the researchers stated. Since the defender controls the communications, they can essentially attempt to exploit weaknesses in the attacker's LLM.

**Counter Attack, Passive Defense**

The Mantis team focused on two types of defensive actions: Passive defenses that attempt to slow the attacker down and raise the cost of their actions, and active defenses that hack back and aim to gain the ability to run commands on the attacker's system. Both strategies were effective with a greater than 95% success rate using the prompt-injection approach, the paper stated.

In fact, the researchers were surprised at how quickly they could redirect an attacking LLM, either causing it to consume resources or even to open a reverse shell back to the defender, says Dario Pasquini, a researcher at GMU and the lead author of the paper.

"It was very, very easy for us to steer the LLM to do what we wanted," he says. "Usually, in a normal setting, prompt injection is a little bit more difficult, but here — I guess because the task that the agent has to perform is very complicated — any kind of injection of prompt, such as suggesting that the LLM do something else, is \[effective\]."

By bracketing a command to the LLM with ANSI characters that hide the prompt text from the terminal, the attack can happen without the knowledge of a human attacker.

**Prompt Injection is the Weakness**

While attackers who want to shore up the resilience of their LLMs can attempt to harden their systems against exploits, the actual weakness is the ability to inject commands into prompts, which is a hard problem to solve, says Giuseppe Ateniese, a professor of cybersecurity engineering at George Mason University.

"We are exploiting those something that is very hard to patch," he says. "The only way to solve it for now is to put some human in the loop, but if you put the human in the loop, then what is the purpose of the LLM in the first place?"

In the end, as long as prompt-injection attacks continue to be effective, Mantis will still be able to turn attacking AIs into prey.

**About the Author**

Veteran technology journalist of more than 20 years. Former research engineer. Written for more than two dozen publications, including CNET News.com, Dark Reading, MIT's Technology Review, Popular Science, and Wired News. Five awards for journalism, including Best Deadline Journalism (Online) in 2003 for coverage of the Blaster worm. Crunches numbers on various trends using Python and R. Recent reports include analyses of the shortage in cybersecurity workers and annual vulnerability trends.

Tag

#intel