By Dmytro Korzhevin, Asheer Malhotra, Vanja Svajcer and Vitor Ventura. 


Cisco Talos has observed a new wave of attacks active since at least late 2023, from a Russian speaking group we track as “UAT-5647”, against Ukrainian government entities and unknown Polish entities. 
UAT-5647 is also known

UAT-5647 targets Ukrainian and Polish entities with RomCom malware variants

PRESS RELEASE

SAN FRANCISCO, Oct. 16, 2024 /PRNewswire/ -- Bugcrowd, the leader in crowdsourced cybersecurity, today released its annual Inside the Mind of a Hacker 2024 report, which analyzed responses from 1,300 hackers, also known as ethical hackers and security researchers on the Bugcrowd Platform. This report provides a comprehensive overview of the hacking community and their perspectives on topics at the forefront of cybersecurity.

AI adoption and integration has continued its rapid momentum within the hacking community. Nevertheless, it continues to pose both benefits and unfortunate cyber risks. According to the report, 82% of hackers believe that the AI threat landscape is evolving too fast to adequately secure.

AI is the new attack vector

This year's report revealed a significant shift in the perceived value of AI in hacking compared to the previous year. While only 21% of hackers believed that AI technologies enhance the value of hacking in 2023, 71% reported it to have value in 2024. Additionally, hackers are increasingly using generative AI solutions, with 77% now reporting the adoption of such tools—a 13% increase from 2023.

While the use and value of AI solutions among hackers have increased, the 2024 report reaffirms that hackers believe AI has limitations. This year's survey revealed that only 22% of hackers believe that AI technologies outperform human hackers, and only 30% believe that AI can replicate human creativity. These results are consistent with those of the 2023 survey.

"There is no denying that AI remains a strong force within the hacking community, changing the very strategies hackers are using to find and report vulnerabilities," says Dave Gerry, CEO of Bugcrowd. "Bugcrowd is in a privileged position to work with a creative, forward-thinking community that thrives on the cutting edge of cybersecurity. Celebrating hackers is part of the core of what we do at Bugcrowd, and these insights can help businesses understand the unique value this community brings to fighting against today's AI-driven cyberattacks."

Key findings from the survey include the following:

*   93% of hackers agree that companies using AI tools have created a new attack vector
    
*   82% believe that the AI threat landscape is evolving too rapidly to be effectively secured from cyberattacks
    
*   86% believe that AI has fundamentally changed their approach to hacking
    
*   74% agree that AI has made hacking more accessible, opening the door for newcomers to join the fold
    
*   Despite these threats, 73% of hackers reported being confident in their ability to uncover vulnerabilities in AI-powered apps
    

These findings point towards the need for hackers in an organization's defense against today's cyberattacks. Although AI is introducing a new attack vector, the majority of hackers still report confidence in their ability to uncover these vulnerabilities, emphasizing the need for organizations to lean on human ingenuity alongside security tooling.

The Rise of Hardware Hacking

The report illuminated the rise of a surprising trend: the increasing prominence of hardware hacking. In the past 12 months, 81% of hardware hackers encountered a new vulnerability they had never seen before, and 64% believe that there are more vulnerabilities now than a year ago. Additionally, in response to the rise of AI, 83% of hardware hackers are now confident in their ability to hack AI-powered hardware and software, indicating a new potential avenue for exploitation. While those familiar with the field may recognize this growing threat, only 33% of hackers in general identified hardware hacking as one of the most valuable specialties. However, there is a low barrier to entry, with 80% of hardware hackers being self-taught.

"Hardware hacking, or the exploitation of vulnerabilities in the physical components of electronic devices, was once considered a specialized field," says Michael Skelton, VP of Security Operations at Bugcrowd. "However, the proliferation of inexpensive, vulnerable smart devices has increased interest in hardware hacking among both ethical hackers and cybercriminals."

A Career Path for a New Generation

This year's survey results also emphasized hacking as a viable and strong career path, particularly for younger generations. Of the respondents, 88% were between the ages of 18 and 34. Additionally, 67% indicated that they are either hacking full-time or actively trying to pursue a full-time hacking career.

Additionally, hacking offers a career path for self-motivated individuals who are eager to learn new skills. While 73% of respondents reported having a college degree or higher, only 29% learned their hacking skills through academic or professional coursework. Instead, 87% reported learning through online resources, 78% through self-study, and 43% through trial and error. Hacking offers younger generations an incredibly desirable career with flexible hours, a remote work environment, and without the requirement of a college degree to achieve success.

Access the Full Report

The survey included 1,300 respondents from 85 countries, including the United States, India, Bangladesh, Pakistan, Nepal, Egypt, Nigeria, the United Kingdom, Vietnam, and Australia. Building upon the success of previous years, this year's edition offers the latest demographic data on the hacking community, a detailed analysis of hackers' daily experiences, and direct insights into hackers' journeys through extensive "Hacker Spotlight" interviews. Readers of this report will better understand how hackers can reduce risks for organizations, provide one of the most significant security returns on investment, and accelerate digital transformation. To download a copy of the Inside the Mind of a Hacker 2024 report, click here.

About Bugcrowd  
We are Bugcrowd. Since 2012, we've been empowering organizations to take back control and stay ahead of threat actors by uniting the collective ingenuity and expertise of our customers and trusted alliance of elite hackers, with our patented data and AI-powered Security Knowledge Platform™. Our network of hackers brings diverse expertise to uncover hidden weaknesses, adapting swiftly to evolving threats, even against zero-day exploits. With unmatched scalability and adaptability, our data and AI-driven CrowdMatch™ technology in our platform finds the perfect talent for your unique fight. We are creating a new era of modern crowdsourced security that outpaces threat actors.

Unleash the ingenuity of the hacker community with Bugcrowd, visit www.bugcrowd.com. Read our blog.

71% of Hackers Believe AI Technologies Increase the Value of Hacking

Threat actors are attempting to abuse the open-source EDRSilencer tool as part of efforts to tamper endpoint detection and response (EDR) solutions and hide malicious activity.
Trend Micro said it detected "threat actors attempting to integrate EDRSilencer in their attacks, repurposing it as a means of evading detection."
EDRSilencer, inspired by the NightHawk FireBlock tool from MDSec, is

Endpoint Security / Malware

Threat actors are attempting to abuse the open-source EDRSilencer tool as part of efforts to tamper endpoint detection and response (EDR) solutions and hide malicious activity.

Trend Micro said it detected "threat actors attempting to integrate EDRSilencer in their attacks, repurposing it as a means of evading detection."

EDRSilencer, inspired by the NightHawk FireBlock tool from MDSec, is designed to block outbound traffic of running EDR processes using the Windows Filtering Platform (WFP).

It supports terminating various processes related to EDR products from Microsoft, Elastic, Trellix, Qualys, SentinelOne, Cybereason, Broadcom Carbon Black, Tanium, Palo Alto Networks, Fortinet, Cisco, ESET, HarfangLab, and Trend Micro.

By incorporating such legitimate red teaming tools into their arsenal, the goal is to render EDR software ineffective and make it a lot more challenging to identify and remove malware.

"The WFP is a powerful framework built into Windows for creating network filtering and security applications," Trend Micro researchers said. "It provides APIs for developers to define custom rules to monitor, block, or modify network traffic based on various criteria, such as IP addresses, ports, protocols, and applications."

"WFP is used in firewalls, antivirus software, and other security solutions to protect systems and networks."

EDRSilencer takes advantage of WFP by dynamically identifying running EDR processes and creating persistent WFP filters to block their outbound network communications on both IPv4 and IPv6, thereby preventing security software from sending telemetry to their management consoles.

The attack essentially works by scanning the system to gather a list of running processes associated with common EDR products, followed by running EDRSilencer with the argument "blockedr" (e.g., EDRSilencer.exe blockedr) to inhibit outbound traffic from those processes by configuring WFP filters.

"This allows malware or other malicious activities to remain undetected, increasing the potential for successful attacks without detection or intervention," the researchers said. "This highlights the ongoing trend of threat actors seeking more effective tools for their attacks, especially those designed to disable antivirus and EDR solutions."

The development comes as ransomware groups' use of formidable EDR-killing tools like AuKill (aka AvNeutralizer), EDRKillShifter, TrueSightKiller, GhostDriver, and Terminator is on the rise, with these programs weaponizing vulnerable drivers to escalate privileges and terminate security-related processes.

"EDRKillShifter enhances persistence mechanisms by employing techniques that ensure its continuous presence within the system, even after initial compromises are discovered and cleaned," Trend Micro said in a recent analysis.

"It dynamically disrupts security processes in real-time and adapts its methods as detection capabilities evolve, staying a step ahead of traditional EDR tools."

Found this article interesting? Follow us on Twitter __ and LinkedIn to read more exclusive content we post.

Hackers Abuse EDRSilencer Tool to Bypass Security and Hide Malicious Activity

Go behind the scenes with Talos incident responders and learn from what we've seen in the field.

Wednesday, October 16, 2024 08:51

Ensuring the cybersecurity of major events — whether it’s sports, professional conferences, expos, inter-government meetings or other gatherings — is a complex and time-intensive task.  

It requires a comprehensive approach and collaboration among various stakeholders, including vendors, hospitality teams, and service providers, to establish a consistent cybersecurity strategy across the entire event ecosystem. 

In our latest version of the “Protecting major events: An incident response blueprint” whitepaper, Cisco Talos Incident Response outlines the essential steps organizations should take to secure any major event. This paper highlights 13 critical focus areas that will guide organizing committees and participating businesses, offering key questions and actionable answers to help ensure robust event security.

Protecting major events: An incident response blueprint

The long-active, India-sponsored cyber-threat group targeted multiple entities across Asia, Africa, the Middle East, and even Europe in a recent attack wave that demonstrated the use of a previously unknown post-exploit tool called StealerBot.

Source: Papilio via Alamy Stock Photo

The elusive, India-based advanced persistent threat (APT) group SideWinder has unleashed a new flurry of attacks against high-profile entities and strategic infrastructure targets that span numerous countries in Asia, the Middle East, Africa, and even Europe, signaling an expansion of its geographic reach. The attacks also show the group is using an advanced post-exploitation toolkit dubbed "StealerBot" to further its cyber-espionage activity, researchers have found.

The state-sponsored group — active since 2012, publicly outed in 2018, and mainly known for attacking rivals in Pakistan, Afghanistan, China, and Nepal — has demonstrated a widening of its geographic scope in the last six months. The latest attacks, observed by researchers at Kaspersky and outlined in a post on the SecureList blog, for the first time revealed some of SideWinder's post-compromise activities, which have remained largely unknown despite years of study by researchers.

Specifically, SideWinder has lately targeted entities in Bangladesh, Djibouti, Jordan, Malaysia, the Maldives, Myanmar, Nepal, Pakistan, Saudi Arabia, Sri Lanka, Turkey, and the United Arab Emirates in the attacks. Affected sectors are varied, and include: government and military entities, logistics, infrastructure and telecommunications companies, financial institutions, universities, and oil trading companies. Attackers also targeted diplomatic entities in Afghanistan, France, China, India, Indonesia, and Morocco.

As for StealerBot, the researchers described the malware — which they believe is the main post-exploitation tool used by SideWinder — as "an advanced modular implant designed specifically for espionage activities."

**SideWinder's Typical Cyberattack Chain**

Though geography and post-exploit tactics vary, SideWinder used its typical attack chain in the latest spate of attacks. The group started with a spear-phishing email with an attachment, which is usually a Microsoft OOXML document — ie, .docx or .xlsx — or a .zip archive, which in turn contains a malicious .lnk file. This file triggers a multistage infection chain with various JavaScript and .NET downloaders, which ultimately ends with the installation of the StealerBot espionage tool for further activity.

The documents used in the spear-phishing part of the campaign often contain information obtained from public websites, "which is used to lure the victim into opening the file and believing it to be legitimate," Kaspersky lead security researchers Giampaolo Dedola and Vasily Berdnikov wrote in the post. In this case, some of the email lures included public photos, images, and references to diplomatic and other activity that might be of interest to the intended target.

All the documents in the attacks use the remote template injection technique to download an .rtf file that is stored on a remote server controlled by the attackers. These files are specifically crafted to exploit CVE-2017-11882, a 7-year-old memory corruption vulnerability in Microsoft Office software, to download further shellcode and malware that uses various tricks to avoid sandboxes and complicate analysis, the researchers said. The ultimate purpose of the malware is to extricate data from infected systems and conduct cyberespionage.

**New StealerBot Modular Malware**

StealerBot, so-named by the attacker, is a modular implant developed with .NET to perform espionage activities. Rather than loading the malware's components on the filesystem of the infected machine, as is typical, the attack chain observed by the researchers loads them into memory by one of the numerous modules of the malware, which in this case acts as a backdoor loader that attackers dubbed "ModuleInstaller."

That module is a downloader that deploys the Trojan that SideWinder uses to maintain a foothold on compromised machines. It's a tool previously wielded by the group and observed by Kaspersky, but not unveiled publicly until now, the researchers noted.

The attackers designed ModuleInstaller to drop at least four files: a legitimate and signed application used to sideload a malicious library; a .config manifest embedded in the program as a resource and required by the next stage to properly load additional modules; a malicious library; and an encrypted payload. "We observed various combinations of the dropped files," the researchers noted.

Another module, called the "Orchestrator," is the main component of the malware that communicates with SideWinder command-and-control (C2) and executes and manages the other malware plugins. All told, StealerBot includes various modules for: installing additional malware, capturing screenshots, logging keystrokes, stealing passwords from browsers, stealing files, phishing Windows credentials, and escalating privileges by bypassing user account control (UAC), among other activities.

**Largely Underestimated Attackers**

SideWinder long has been perceived as a low-skilled threat group due to its use of public exploits and remote access Trojans (RATs), as well as malicious .lnk files and scripts as infection vectors, according to Kaspersky. However, they should not be underestimated by defenders, as "their true capabilities only become apparent when you carefully examine the details of their operations," the researchers wrote.

As the new wave of attacks shows "a significant expansion of the group’s activities," those who may be targeted should be on alert and aware of the threat posed by the group, they said.

To help defenders recognize the presence of SideWinder and its tool StealerBot on their networks, the researchers included a comprehensive list of indicators of compromise (IoCs) for various stages of the attack in their post.

The IoCs include references to malicious documents, and .rtf and .lnk files, as well as specific IoCs to various modules of StealerBot. A long list of malicious domains and IPs associated with the attacks also is included in the post.

**About the Author**

Elizabeth Montalbano is a freelance writer, journalist, and therapeutic writing mentor with more than 25 years of professional experience. Her areas of expertise include technology, business, and culture. Elizabeth previously lived and worked as a full-time journalist in Phoenix, San Francisco, and New York City; she currently resides in a village on the southwest coast of Portugal. In her free time, she enjoys surfing, hiking with her dogs, traveling, playing music, yoga, and cooking.

Sidewinder Casts Wide Geographic Net in Latest Attack Spree

Organizations should be on high alert until next month's US presidential election to ensure the integrity of the voting process, researchers warn.

Source: Jon Helgason via Alamy Stock Photo

Cyber-threat actors have ramped up their targeting of the 2024 US elections with a flood of malicious activity expected to peak over the next month, aimed at causing disruption to voters and the election process and requiring increased vigilance on the part of stakeholders.

Specifically, attackers have bolstered election-related threat activity since the beginning of the year with an increase in the sale of phishing kits targeting US voters and campaign donors; the registration of more than 1,000 domains aimed at exploiting election-related content for malicious purposes; and increased ransomware activity targeting government entities, according to research from FortiGuard Labs Threat Research released today.

Since the inception of Internet-related threats, cyber-threat actors have typically increased malicious activity ahead of elections, notes Derek Manky, chief security strategist and vice president of global threat intelligence at Fortinet. However, they aim to be especially disruptive during the current election cycle, requiring that all stakeholders be prepared to fend off malicious actors in the upcoming weeks to protect election outcomes.

"As the 2024 US presidential election approaches, it's critical to recognize and understand the cyber threats that may impact the integrity and trustworthiness of the election process and the welfare of the participating citizens," he says.

Indeed, separate research has found that adversaries from Russia, China, and Iran in particular have been using cyber operations to stoke discord and influence election outcomes rather than make direct attacks on voting machines or other voter infrastructure. These more insidious tactics require a different type of vigilance on the part of defenders, the researchers noted.

**Specific Threats to Watch For**

FortiGuard Labs' latest election-threat research is the result of analysis of threats gathered from January 2024 to August 2024 that may affect US-based entities and the electoral process. The researchers discovered several key areas of threat activity that have been on the rise.

One is a significant increase in the availability of affordable phishing kits on the Dark Web designed to target voters and donors by impersonating the presidential candidates and their campaigns. Specifically, the researchers found kits for $1,260 created to impersonate US presidential candidates and to harvest personal information, including names, addresses, and credit card details.

Part of the phishing activity around the current election cycle also includes an increase of highly convincing mobile scams that use phone calls, voicemails, or messaging services that leverage deepfake technology to spread misinformation, which can affect voter outcomes, notes Alex Quilici, CEO at YouMail.

"AI can now create highly convincing voice attacks that make it sound like a trusted figure, such as a candidate, urging you not to vote or spreading false information," he says. "This kind of deception can seriously undermine public trust and disrupt the electoral process."

Attackers also have registered more than 1,000 new potentially malicious domains since the beginning of 2024 that incorporate election-related content and candidates to lure unsuspecting targets and potentially conduct nefarious activities, the researchers noted. The two most-used hosting providers for these election-themed websites are AMAZON-02 and CLOUDFLARENET, demonstrating that attackers are leveraging known, reputable services to bolster the legitimacy of malicious domains.

Another way cyberattackers can spread misinformation and disrupt the democratic process is through the use of people's personal information to directly target them, the researchers noted. Fortinet found that there currently is an abundance of this type of material on the Dark Web, with more than 1.3 billion rows of combo lists — which include usernames, email addresses, and passwords — of US citizens for sale for nefarious use.

The availability of this data poses a considerable risk for credential-stuffing attacks that allow cybercriminals to gain unauthorized access to people's accounts. Overall, the availability of so much personal data of various election stakeholders creates potential indirect disruption in the voting process, notes Casey Ellis, founder and chief strategy officer at Bugcrowd.

"While it may be difficult to use these records to commit the kind of fraud or attacks that would directly modify the outcome of an election, it's certainly a cheap and simple exercise to simply highlight the possibility of their use as a way to instill distrust in the democratic process, and to potential affect and manipulate voter turnout," he says.

FortiGuard Labs researchers also noted a 28% increase in ransomware attacks against the US government year-over-year based on observed leak sites. This type of activity also can threaten the integrity of the election process by undermining citizens' trust in the ability of the government to protect the personal data they collect from them.

**Protect Election Integrity**

To ensure the US presidential election process runs smoothly for all that wish to participate, Fortinet offered some recommendations to prevent and mitigate attacks between now and election day. The researchers advised that individuals and organizations alike always remain vigilant for suspicious behavior or activity leading up to major election-related events and prioritize good cyber hygiene in general to reduce potential threats.

Organizations, especially those related to the election or government agencies, should prioritize employee training and awareness about the cyber threats that exist that aim to disrupt the election process. Enforcing multifactor authentication and a strong password policy across both individuals' and organizations' online accounts also can protect against intrusion.

Finally, any organization with a stake in the election also should install endpoint protection solutions, patch operating systems and Web servers, and update software regularly to ensure systems are as secure as possible, Fortinet recommended.

**About the Author**

Elizabeth Montalbano is a freelance writer, journalist, and therapeutic writing mentor with more than 25 years of professional experience. Her areas of expertise include technology, business, and culture. Elizabeth previously lived and worked as a full-time journalist in Phoenix, San Francisco, and New York City; she currently resides in a village on the southwest coast of Portugal. In her free time, she enjoys surfing, hiking with her dogs, traveling, playing music, yoga, and cooking.

Cyberattackers Unleash Flood of Potentially Disruptive Election-Related Activity

Intel Broker claims a major data breach at Cisco, allegedly stealing source codes, confidential documents, and credentials from…

Intel Broker claims a major data breach at Cisco, allegedly stealing source codes, confidential documents, and credentials from global firms like Verizon, AT&T, Microsoft, and more. Data is now for sale on Breach Forums.

Intel Broker, a hacker notorious for high-profile data breaches, is claiming to have breached the technology giant Cisco Systems, Inc. In a post on the cybercrime platform **Breach Forums**, the hacker stated that the breach enabled them to steal a massive amount of sensitive information from Cisco’s systems.

According to the hacker, the alleged data breach took place on October 10, 2024, while the Breach Forum post was published earlier today on October 14, 2024.

Intel Broker’s post (Screenshot: Hackread.com)

****What Was Allegedly Stolen?****

As seen by the Hackread.com research team, Intel Broker has listed a massive amount of data that was allegedly stolen in the breach, including:

*   **Source Code**: Projects from GitHub, GitLab, and SonarQube, critical to Cisco’s development efforts.
*   **Hard-Coded Credentials**: Sensitive information like login details embedded in source code.
*   **Certificates and Keys**: SSL certificates, and public and private keys crucial for secure communications.
*   **Confidential Documents**: Internal documents and information classified as “Cisco Confidential.”
*   **API Tokens and Storage Buckets**: AWS private buckets, Azure storage buckets, and API tokens that could be used to access critical systems.
*   **Other Sensitive Information**: Jira tickets, Docker builds, and Cisco premium products are also listed.

****Impact on Major Corporations****

Intel Broker also shared a list of companies whose production source codes were allegedly taken during the breach. The list includes several high-profile firms, particularly in the telecommunications and financial sectors, such as:

*   **Telecom Companies**: Verizon, AT&T (USA and Mexico), British Telecom, T-Mobile (USA and Poland), Vodafone (Albania and Australia), and Turkcell.
*   **Financial Institutions**: Bank of America, Barclays, and National Australian Bank.
*   **Tech and Health**: Microsoft, Liberty Global, and Dignity Health.

Sample documents shared by the hacker (Screenshot: Hackread.com)

****Data for Sale****

Intel Broker is offering the stolen data for sale in exchange for Monero (XMR), a cryptocurrency known for its privacy features. The hacker indicated that they are open to using a middleman to facilitate the transaction, ensuring **anonymity** for both the buyer and seller. This method is a common practice among cybercriminals to avoid detection and tracking by authorities.

****Unverified but Serious Claims****

At the time of writing, Hackread.com, which first spotted the hacker’s claims, has reached out to Cisco for comment, but no official response has been given. The breach, if confirmed, could have major consequences for Cisco and the affected companies, raising concerns about the extent of the damage and the potential exploitation of the compromised data.

****Intel Broker and Previous Breaches****

Intel Broker is known for high-profile data breaches. **In June 2024**, the hacker claimed to have breached Apple Inc., stealing source code for internal tools. The same hacker boasted about **breaching AMD** (Advanced Micro Devices, Inc.), and stealing employee and product information.

**In May 2024**, Intel Broker hacked Europol, a breach that the agency later confirmed. Some of the hacker’s previous data breaches are listed below:

*   **Tech in Asia**
*   **Space-Eyes**
*   **Home Depot**
*   **Facebook Marketplace**
*   **U.S. contractor Acuity Inc.**
*   **Staffing giant Robert Half**
*   **Los Angeles International Airport**
*   **Alleged breaches of HSBC and Barclays Bank**

Although the hacker’s origins and affiliates are unknown, according to the United States government, IntelBroker is alleged to be the perpetrator behind one of the **T-Mobile data breaches**.

Nevertheless, these claims regarding the Cisco data breach go on to show the cybersecurity risks faced even by large organizations. As more details emerge, the scale of this breach and its potential fallout will be closely watched.

1.  **Akira Ransomware Targets Businesses via Exploited CISCO VPNs**
2.  **Cisco Network Breach as Employee’s Google Account** **was** **Hacked**
3.  **Hackers Claim 10TB Breach at Russian Cybersecurity Firm Dr.Web**
4.  **Hackers leave US flag after targeting Cisco switches in Russia & Iran**
5.  **Ex-worker hacked Cisco AWS Infrastructure; erased virtual machines**

Intel Broker Claims Cisco Breach, Selling Stolen Data from Major Firms

Organizations are grappling with the risks of having outdated hardware handling core workloads, mission-critical applications no one knows how to update or maintain, and systems that IT and security teams don't know about.

Dave Lewis, Global Advisory CISO, 1Password

October 14, 2024

3 Min Read

Source: Aleksey Popov via Alamy Stock Photo

COMMENTARY

When I began my security career, everything was an adventure — new technologies, new opportunities, and new lessons to learn. Some of those lessons have stayed with me over the years. Simple on the surface, these lessons have had a significant impact and proved valuable over time. Yet, when I look at the wider industry, I often find myself vexed at the current state of affairs. 

**The Beige Desktop Is Everywhere**

The best example of this flustered feeling is the pervasive nature of the "beige desktop." We have all seen them in our industry travels — machines that predate many of the technologies we rely on today. Hardware that soldiers on from the dark recesses of a data center's raised floor. 

You can see where this is heading. That system is invariably running code written by a summer student long ago and has now become mission-critical. Code that was not properly commented or documented. An application that has somehow become indispensable to the business. 

How does this keep happening? I've often pondered this question. Whenever I bring it up at conferences, heads always nod in understanding. Those systems that lurk in the shadows of a data center. 

**Hard to Get Rid of Shadows**

We often hear the term "shadow IT." It usually finds its way into conversations with a sense of derision. A few months ago, I was giving a talk at a conference when I asked the audience if they had encountered the beige desktop in their environments. The audience laughed, grimaced, and hung their heads — confirming my thoughts. I paused and asked how many companies present had controls in place in their environments for shadow IT. Every hand went up. 

I let the question hang in the air for a moment. Then I asked the audience a follow-up query: "How many of you here have shadow IT in your environments?" There was some hesitation. Eyes darted around nervously. Slowly but surely, every hand went up again. 

We had an interesting conversational moment. These companies all had controls in place to guard against shadow IT, yet … it still existed. We had discovered Schrödinger's IT security problem. It simultaneously exists and doesn't. 

**Who Owns the Risk?This begs the question: Who truly owns the risk of shadow IT? While the knee-jerk reaction might be to assign this to the chief information security officer, I wonder if that is fair. The CISO puts security controls in place. The CISO ensures that there are policies and procedures around handling the risks presented by shadow IT. But it continues. Is it fair to say the CISO is responsible at that point? Just thinking out loud. Could this risk be more appropriately assigned to the chief financial officer, as it presents a potential material enterprise risk so thereby is this executive's responsibility? I would love to see this develop into a broader conversation because, honestly, I'm unsure of the answer and would love the input from the CISO community. ****How We Wound Up Here**

Shadow IT rarely, if ever, originates from a place of malice. These projects are quite often built to satisfy the need for innovation. Other examples of why this happens could include the perceived inadequacy of the deployed systems that support development in the enterprise. Or it simply occurs out of a need for speed and convenience. 

It's often easier to ask for forgiveness than permission. While the beige desktop may be a tongue-in-cheek story, it serves as an example of what happens in environments across the globe. 

**Top Dead Center**

How do we move toward an enterprise or SMB environment that supports innovation while remaining safe and secure? There is a need to provide visibility and security to deal with tools and projects that may not have been vetted or approved by the IT and security teams. 

It's time to move away from the beige desktops and toward a technological engine that empowers businesses to drive innovation safely and securely. 

**About the Author**

Global Advisory CISO, 1Password, 1Password

Dave is the Global Advisory CISO at 1Password. He brings over 30 years of industry experience, extensively in IT security operations and management, at companies such as Akamai, IBM, Duo Security, Cisco, and AMD. He is also the founder of the security site Liquidmatrix Security Digest as well as host of the Liquidmatrix, Plaintext, and Chasing Entropy podcasts. Dave currently serves on the board of directors for BSides Las Vegas and the advisory board for the Black Hat Sector Security Conference. He co-founded the BSides Toronto conference and was a goon on the speaker operations team for DEF CON for over 13 years. He previously held a board position at (ISC)². For fun, Dave loves playing bass guitar, grilling, and spending quality time with his kids. He’s also a part owner of a whisky distillery and a soccer team.

The Lingering 'Beige Desktop' Paradox

Attackers can introduce a malicious document in systems such as Microsoft 365 Copilot to confuse the system, potentially leading to widespread misinformation and compromised decision-making processes.

Source: Mopic via Shutterstock

Attackers can add a malicious document to the data pools used by artificial intelligence (AI) systems to create responses, which can confuse the system and potentially lead to misinformation and compromised decision-making processes within organizations.

Researchers from the Spark Research Lab at the University of Texas (UT) at Austin discovered the attack vector, which they've dubbed ConfusedPilot because it affects all retrieval augmented generation (RAG)-based AI systems, including Microsoft 365 Copilot. This includes other RAG-based systems that use Llama, Vicuna, and OpenAI, according to the researchers.

"This attack allows manipulation of AI responses simply by adding malicious content to any documents the AI system might reference," Claude Mandy, chief evangelist at Symmetry, wrote in a paper about the attack, which was presented at the DEF CON AI Village 2024 conference in August but was not widely reported. The research was conducted under the supervision of Symmetry CEO and UT professor Mohit Tiwari.

Given that 65% of Fortune 500 companies currently implement or are planning to implement RAG-based AI systems, the potential impact of these attacks cannot be overstated," Mandy wrote. Moreover, the attack is especially dangerous that it requires only basic access to manipulate responses by all RAG-based AI implementations, can persist even after malicious content is removed, and bypasses current AI security measures, he said.

**Malicious Manipulation of RAG**

RAG is a technique for improving response quality and eliminating a large language model (LLM) system’s expensive retraining or fine-tuning phase. It adds a step to the system in which the model retrieves external data to augment its knowledge base, thus enhancing accuracy and reliability in generating responses without the need for retraining or fine-tuning, the researchers said.

The researchers chose to focus on Microsoft 365 Copilot for the sake of their presentation and their paper, even though it is not the only RAG-based system affected. Rather, "the main culprit of this problem is misuse of RAG-based systems … via improper setup of access control and data security mechanisms," according to the ConfusedPilot website hosted by the researchers.

In normal circumstances, a RAG-based AI system will use a retrieval mechanism to extract relevant keywords to search and match with resources stored in a vector database, using that embedded context to create a new prompt containing the relevant information to reference.

**How the Attack Works**

In a ConfusedPilot attack, a threat actor could introduce an innocuous document that contains specifically crafted strings into the target’s environment. "This could be achieved by any identity with access to save documents or data to an environment indexed by the AI copilot," Mandy wrote.

The attack flow that follows from the user's perspective is this: When a user makes a relevant query, the RAG system retrieves the document containing these strings. The malicious document contains strings that could act as instructions to the AI system that introduce a variety of malicious scenarios.

These include: content suppression, in which the malicious instructions cause the AI to disregard other relevant, legitimate content; misinformation generation, in which the AI generates a response using only the corrupted information; and false attribution, in which the response may be falsely attributed to legitimate sources, increasing its perceived credibility.

Moreover, even if the malicious document is later removed, the corrupted information may persist in the system’s responses for a period of time because the AI system retains the instructions, the researchers noted.

**Victimology and Mitigations**

The ConfusedPilot attack basically has two victims: The first is the LLM within the RAG-based system, while the second is the person receiving the response from the LLM, who very likely could be an individual working at a large enterprise or service provider. Indeed, these two types of companies are especially vulnerable to the attack, as they allow multiple users or departments to contribute to the data pool used by these AI systems, Mandy noted.

"Any environment that allows the input of data from multiple sources or users — either internally or from external partners — is at higher risk, given that this attack only requires data to be indexed by the AI Copilots," he wrote.

Enterprise systems likely to be negatively affected by the attack include enterprise knowledge-management systems, AI-assisted decision support systems, and customer-facing AI services.

Microsoft did not immediately respond to request for comment by Dark Reading on the attack's affect on Copilot. However, the researchers noted in their paper that the company has been responsive in coming up with "practical mitigation strategies" and addressing the potential for attack in its development of its AI technology. Indeed, the latter is key to long-term defense against such an attack, which depends on "better architectural models" that "try to separate the data plan from the control plan in these models," Mandy noted.

Meanwhile, current strategies for mitigation include: data access controls that limit and scrutinize who can upload, modify, or delete data that RAG-based systems reference; data integrity audits that regularly verify the integrity of an organization's data repositories to detect unauthorized changes or the introduction of malicious content early; and data segmentation that keeps sensitive data isolated from broader datasets wherever possible to prevent the spread of corrupted info across the AI system.

**About the Author**

Elizabeth Montalbano is a freelance writer, journalist, and therapeutic writing mentor with more than 25 years of professional experience. Her areas of expertise include technology, business, and culture. Elizabeth previously lived and worked as a full-time journalist in Phoenix, San Francisco, and New York City; she currently resides in a village on the southwest coast of Portugal. In her free time, she enjoys surfing, hiking with her dogs, traveling, playing music, yoga, and cooking.

ConfusedPilot Attack Can Manipulate RAG-Based AI Systems

PRESS RELEASE

SAN FRANCISCO, Oct. 10, 2024 /PRNewswire/ -- Relyance AI, the leading AI-powered data governance platform that provides complete visibility and control over enterprise-wide data, today announced a $32.1 million Series B funding round to scale operations and meet the needs of the exploding use of artificial intelligence in the enterprise. Thomvest Ventures led the round. M12, Microsoft Ventures Fund, also participated in addition to Cheyenne Ventures and existing investors Menlo Ventures and Unusual Ventures.

As demand for AI surges in the enterprise, global regulators are mandating data protection safeguards that companies find impossible to implement. At the same time, legal, security, and engineering teams are grappling with endless questions about how customer data is being used to train AI models. The impact? Executives are stifled by their inability to realize AI's innovation potential, and according to KPMG, more than two-thirds (76%) of enterprises are worried about data privacy and security when they partner with third parties. The result could be catastrophic: more than a quarter of the Fortune 500 identified AI regulation as a risk in annual reports to the SEC, and many continue to struggle with established data privacy regulations such as GDPR, which led to a record €2.1 billion in fines in 2023.

This has become a business-critical problem for all enterprises that was impossible to solve before Relyance AI's fully integrated governance platform. Until now, privacy and security have been seen as separate challenges, with one woefully unaware of the other. Privacy teams didn't know if their commitments to regulations and customers were being met, and security teams didn't know what data should be in AI models. Relyance AI marries the two into one joint solution, which is the only way to enable innovation while ensuring compliance in a rapidly evolving regulatory landscape.

"The era of accepting subpar privacy, DSPM, and AI governance solutions is over. Relyance AI sets a new standard where data protection and innovation are not mutually exclusive," said Abhi Sharma, CEO and co-founder of Relyance AI. "It's impossible to keep up with the current state of regulations, especially when GDPR, HIPAA, the EU's AI Act, and a mosaic of local U.S. privacy laws are all different and sometimes at odds. We're making it possible to demystify this and embolden the C-suite, engineers, and legal teams to urgently green-light AI in the enterprise with an integrated governance approach." 

Relyance AI safeguards businesses from fines and reputation damage while boosting customer trust to accelerate business growth. The platform provides complete visibility into enterprise-wide data processing and compares it against contractual commitments, global privacy regulations, and compliance frameworks. The company has scaled significantly to meet recent demand for these capabilities. In the first half alone, Relyance AI increased its enterprise customer base by 30%, and the company is projected to double its annual recurring revenue in 2024. Its client list now includes Coinbase, Fivetran, Verkada, Snowflake, Logitech, Plaid, and Notion.

"We're thrilled to lead the investment in Relyance AI, a unique, automated data governance platform that addresses the urgent need for real-time data visibility and compliance in a world of explosive data growth and emerging regulations," said Umesh Padval, Managing Director, Thomvest Ventures. "Relyance AI empowers Chief Privacy, Security, and Information Officers to manage data privacy and compliance, avoiding costly penalties while driving safe and responsible AI adoption. We are excited to partner with CEO Abhi Sharmaand his team, who have built a solution that transforms compliance into a competitive advantage, enabling businesses to scale AI with trust and transparency."

"The future of AI governance isn't about compliance alone – it's about building trust, transparency, and accountability into every layer of your technology," said Todd Graham, Managing Partner at M12. "Relyance AI is the catalyst for that transformation, paving the way for AI implementation that is both compliant and incredibly fast."

"With Relyance AI, we established enhanced visibility of data processing activities, seeing an impressive increase of 1,660% within three weeks of deployment," said Deborah Usry, Senior Privacy and Product Counsel at NextRoll. "What used to be a time-consuming manual process is now an automated task that produces a robust record of our processing activities."

The funding will further develop Relyance AI's platform and scale its go-to-market efforts in response to significant recent momentum. Learn more here.

About Relyance AI  
Relyance AI is the new standard for organizations trust and data governance infrastructure. Relyance AI is the only platform providing real-time visibility into how and where data is being used compared to customer agreements, global privacy regulations, and compliance frameworks. Companies of all industries and sizes – including Coinbase, Snowflake, Logitech, Plaid, Notion, and more – trust Relyance AI to safeguard businesses from fines and reputational damage and to deliver the most complete customer trust experiences.

Tag

#cisco