Critical Vulns Found in Ray Open Source Framework for AI/ML Workloads

Source: Ken stocker via Shutterstock

Organizations using Ray, the open source framework for scaling artificial intelligence and machine learning workloads, are exposed to attacks via a trio of as yet unpatched vulnerabilities in the technology, researchers said this week.

Potentially Heavy Damage

The vulnerabilities give attackers a way to, among other things, gain operating system access to all nodes in a Ray cluster, enable remote code execution, and escalate privileges. The flaws present a threat to organizations that expose their Ray instances to the Internet or even a local network.

Researchers from Bishop Fox discovered the vulnerabilities and reported them to Anyscale — which sells a fully managed version of the technology — in August. Researchers from security vendor Protect AI also privately reported two of the same vulnerabilities to Anyscale previously.

But so far, Anyscale has not addressed the flaws, says Berenice Flores Garcia, senior security consultant at Bishop Fox. “Their position is that the vulnerabilities are irrelevant because Ray is not intended for use outside of a strictly controlled network environment and claims to have this stated in their documentation,” Garcia says.

Anyscale did not immediately respond to a Dark Reading request for comment.

Ray is a technology that organizations can use to distribute the execution of complex, infrastructure-intensive AI and machine learning workloads. Many large organizations (including OpenAI, Spotify, Uber, Netflix, and Instacart) currently use the technology for building scalable new AI and machine learning applications. Amazon’s AWS has integrated Ray into many of its cloud services and has positioned it as technology that organizations can use to accelerate the scaling of AI and ML apps.

Easy to Find and Exploit

The vulnerabilities that Bishop Fox reported to Anyscale pertain to improper authentication and input validation in Ray Dashboard, Ray Client, and potentially other components. The vulnerabilities affect Ray versions 2.6.3 and 2.8.0 and allow attackers a way to obtain any data, scripts, or files stored in a Ray cluster. “If the Ray framework is installed in the cloud (i.e., AWS), it is possible to retrieve highly privileged IAM credentials that allow privilege escalation,” Bishop Fox said in its report.

The three vulnerabilities that Bishop Fox reported to Anyscale are CVE-2023-48023, a remote code execution (RCE) vulnerability tied to missing authentication for a critical function; CVE-2023-48022, a server-side request forgery vulnerability in the Ray Dashboard API that enables RCE; and CVE-2023-6021, an insecure input validation error that also enables a remote attacker to execute malicious code on an affected system.

Bishop Fox’s report on the three vulnerabilities included details on how an attacker could potentially exploit the flaws to execute arbitrary code.

The vulnerabilities are easy to exploit, and attackers do not require a high level of technical skills to take advantage of them, Garcia says. “An attacker only requires remote access to the vulnerable component ports — ports 8265 and 10001 by default — from the Internet or from a local network,” and some basic Python knowledge, she says.

“The vulnerable components are very easy to find if the Ray Dashboard UI is exposed. This is the gate to exploit the three vulnerabilities included in the advisory,” she adds. According to Garcia, if the Ray Dashboard is not detected, a more specific fingerprint of the service ports would be required to identify the vulnerable ports. “Once the vulnerable components are identified, they are very easy to exploit following the steps from the advisory,” Garcia says.

Bishop Fox’s advisory shows how an attacker could exploit the vulnerabilities to obtain a private key and highly privileged credentials from an AWS cloud account where Ray is installed. But the flaws affect all organizations that expose the software to the Internet or local network.

Controlled Network Environment

Though Anycase did not respond to Dark Reading, the company’s documentation states the need for organizations to deploy Ray clusters in a controlled network environment. “Ray expects to run in a safe network environment and to act upon trusted code,” the documentation states. It mentions the need for organizations to ensure that network traffic between Ray components happens in an isolated environment and to have strict network controls and authentication mechanisms when accessing additional services.

“Ray faithfully executes code that is passed to it — Ray doesn’t differentiate between a tuning experiment, a rootkit install, or an S3 bucket inspection,” the company noted. “Ray developers are responsible for building their applications with this understanding in mind.”

About the Author(s)

Jai Vijayan is a seasoned technology reporter with over 20 years of experience in IT trade journalism. He was most recently a Senior Editor at Computerworld, where he covered information security and data privacy issues for the publication. Over the course of his 20-year career at Computerworld, Jai also covered a variety of other technology topics, including big data, Hadoop, Internet of Things, e-voting, and data analytics. Prior to Computerworld, Jai covered technology issues for The Economic Times in Bangalore, India. Jai has a Master’s degree in Statistics and lives in Naperville, Ill.