Buggy CrowdStrike EDR Update Crashes Windows Systems Worldwide

Source: SOPA Images Limited via Alamy Stock Photo

This is a breaking news story and will be updated as new developments occur.

This morning, Microsoft servers across the world displayed the dreaded “blue screen of death,” leading to mass IT outages that disrupted business, airlines and flights, healthcare providers, banks, and more. The cause: A defective update to CrowdStrike Falcon Sensor, a widely used cloud-based endpoint detection and prevention (EDR) software program.

CrowdStrike said its engineering team has identified the issue that caused the massive disruption to Windows-based systems: A bug in the Memory Scanning prevention policy, which was not identified during their testing stages, Callie Guenther, senior manager at Critical Start, noted in an emailed statement.

“While CrowdStrike likely performed standard regression and functionality tests, these were insufficient because they did not simulate the real-world deployment environment where the bug caused the Falcon sensor to consume 100% of a CPU core,” she wrote. This ultimately led to system performance issues.

CrowdStrike has since reverted the flawed Falcon software update. Even so, some users are still experiencing system crashes or are unable to stay online to receive the new and fixed version. The cybersecurity vendor has provided workaround steps for this issue.

In a post on social platform X, Microsoft CEO Satya Nadella said the company is aware of the issue and is working closely with CrowdStrike to provide technical support to its customers and get their systems back online.

Falcon Fallout

The severity of the broken CrowdStrike update became increasingly painful as victim reports rolled in throughout the day: More than 1,300 flights have been canceled or delayed, trains, card payments in stores, pharmacies, and even general practitioner (GP) surgeries were stalled.

The Department of Health in Belfast reported that two-thirds of GP practices in Northern Ireland have been affected, with patient records inaccessible as well as lab tests and routine prescriptions.

Delta flights have been paused as it “works through a vendor technology issue,” the New York Times reported, and Turkish Airlines has canceled at least 84 flights. Employees at financial institutions like JPMorgan Chase and Instinet have had trouble accessing their corporate systems as operations began to stutter.

Even the Paris Olympics organizing committee reports that its IT operations have been affected, mainly affecting delivery of uniforms and accreditations.

Meanwhile, President Joe Biden has been briefed on the outage, according to the White House, and administration officials are reportedly in touch with affected entities as well as CrowdStrike, which is working with customers that have been impacted.

“Mac and Linux hosts are not impacted,” George Kurtz, president and CEO of CrowdStrike, wrote online. “This is not a security incident or cyberattack. The issue has been identified [and isolated,] and a fix has been deployed. We refer customers to the support portal for the latest updates and will continue to provide complete and continuous updates on our website.”

It’s Not a Data Breach, but it’s a Disaster

In an industry where cybersecurity practices and services are meant to protect an enterprise without interrupting them, this outage proves that “even non-malicious cybersecurity failures can bring businesses to their knees,” according to Maxine Holt, cybersecurity analyst at Omdia.

This massive incident underscores an over-reliance on cloud services, Holt noted in an online statement, and the outage may prompt organizations to reconsider moving their mission-critical applications to the cloud.

“Omdia’s Cloud and Data Center analysts have long warned about over-reliance on cloud services,” Holt said. “Today’s outages will make enterprises rethink moving mission-critical applications off-premises. The ripple effect is massive, hitting CrowdStrike, Microsoft, AWS, Azure, Google, and beyond. CrowdStrike’s shares have plummeted by more than 20% in unofficial pre-market trading in the US, translating to a staggering $16 billion loss in value.”

As CrowdStrike will undoubtedly face scrutiny as it gets back on its feet, only time will tell how this outage could affect regulation and pressure on software vendors.

“We need stronger regulations and guidance on vendor responsibilities for functional testing,” Josh Thorngren, security strategist at ForAllSecure, wrote in an emailed statement. “If you’re not testing the behavior of your application under-expected (and unexpected) conditions with every update — this type of issue will always be a risk.”