Finding the next Log4j – OpenSSF’s Brian Behlendorf on pivoting to a ‘risk-centred view’ of open source development

Apache pioneer says ‘use at your own risk’ model no longer tenable as OpenSSF ramps up end user engagement

The Open Source Security Foundation (OpenSSF) recently adopted Microsoft’s Secure Supply Chain Chain Consumption Framework (S2C2F) to help reduce vulnerabilities in open source software – a once unthinkable partnership given Redmond’s former animosity to open source.

Owner of GitHub since 2016 and a major Linux contributor, Microsoft developed S2C2F in 2019.

The threat-based risk-reduction framework lists supply chain threats to open source software, and provides organizations with guides for assessing their maturity and implementing the framework’s requirements.

The move is just one step OpenSSF is taking to tackle a surge in open source vulnerabilities and sevenfold rise in attacks against the ecosystem.

Varying degrees of rigor

For OpenSSF general manager Brian Behlendorf, adopting S2C2F is one highlight of his first year in the post, during which OpenSSF has surpassed 100 members.

Behlendorf is a co-creator of Apache HTTP Server, former CTO of the World Economic Forum, and member of Linux Foundation – which runs the OpenSSF project – since 2014.

“The OpenSSF is a collection of initiatives that aim to make open source software more secure, to more quickly find vulnerabilities, to find ways to get those vulnerabilities fixed more quickly… but also to really address the emerging weaknesses in the global software supply chain,” Behlendorf tells The Daily Swig.

“Frankly, with 70 to 90% of software in any given commercial package being pre-existing open source code, there is no distinction anymore between the world of open source and the world of commercial software,” he suggests.

MUST READ ‘We don’t teach developers how to write secure software’ – Linux Foundation’s David A Wheeler on reversing the CVE surge

Behlendorf also points to the proliferation of open source code through environments such as Kubernetes, and the way this is exposing software components that might not all have been built “to the same degree of rigor.

“Open source software tends to have a pretty decent reputation for security thanks to marquee projects like Linux, where the amount of investment into hardening and into testing and verifying and all that kind of stuff is pretty high,” he says.

“And many other projects have a high degree of diligence when it comes to the integrity and security of code. But not all projects achieve that. And a lot of software, especially the smaller modules, are built frankly by people on the way to doing something else.”

He continues: “They put it up on GitHub, or they throw a package on NPM. They wake up one day and thousands or millions of people are using their package without really thinking about it, or having any way to measure whether one package is more rigorously built and inspected and reviewed than another.”

Lessons of Log4j

Part of the OpenSSF’s mission, then, is to improve security practices among developers. This goes hand in hand with helping organizations that use open source code to assess security risks.

“The focus is towards objective measures of risk and ways to programmatically help consumers be smarter about their choices of software, and [for] developers realizing that there’s more that they can do to protect their end users,” says Behlendorf.

This has become all the more important since Log4j. “What we’ve found over the last year is, especially in the wake of Log4j, is a whole lot of folks want to understand better how to systematically reduce risk in the global supply chains. How do we do reduce the chances of the next Log4j?”

This has to be done, he says, to reduce the impact of a vulnerability on the millions of businesses globally that rely on open source software. New processes for software development, more scrutiny of code and software components, better use of software bills of materials (SBOMs), and developer education will all help.

“We can’t say what the next Log4j will be, but we can say which are the 200 packages that have a 1% chance of being the next Log4j,” he says.

RELATED Third of Log4j downloads still pull vulnerable version despite threat of supply chain attacks

Much of the code in use now – both commercial and open source – was not designed to resist today’s threats. The challenge is identifying those vulnerable components, and fixing or mitigating any vulnerabilities. A relatively small increase in investment in security research could, Behlendorf argues, save “billions of dollars of potential impact.

“We’ve been very poor at applying risk to software development,” he says. “It’s all been caveat emptor and good luck. Hopefully we are part of the pivot towards taking a more risk-centered view of how to build critical digital infrastructure.”

Hesitancy over security updates

One way of doing this is through the OpenSSF scorecard, which developers can use to check for vulnerabilities affecting source code, build, dependencies, testing, and project maintenance. It’s not foolproof, Behlendorf concedes, but enables developers to take security seriously.

But the industry, and especially developers in end user businesses, face other hurdles. Behlendorf highlights hesitancy around deploying fixes, even when patching known vulnerabilities. The fear is of disrupting a critical system.

Industry needs to convince CIOs “that they can make that update without it having cascading effects on the rest of their infrastructure,” he says. “Many organizations have been burned by doing frequent updates and discovering that that APIs change and old stuff doesn’t work.” To address this, developers need to make their code more forward-compatible, as well as automating the process of updates.

Nor do organizations have effective ways of measuring technical debt, and quantifying the security risks posed by outdated code. “Not just open source software, but all commercial software, has always been ‘use at your own risk’,” he explains. And that “no liability” model has been essential for open source ecosystem to grow.

Partnering with Microsoft

As a result, software distributors and end users need better tools to identify and reduce risk. This is why OpenSSF has worked with Microsoft to use the S2C2F, despite the acrimonious past between Richmond and open source community.

“That was a long time ago,” says Behlendorf. “Today Microsoft is a major contributor to the Linux kernel. They are heavy spenders on improvements to lots of different open source packages.”

S2C2F will, OpenSSF believes, put more rigor into software development, and should supplement other initiatives such as the SLSA framework.

The S2C2F will now have its own Special Initiative Group at OpenSSF that, Behlendorf says, will open it up to other vendors and stakeholders.

“Now it’s an OpenSSF project, we expect to see participation from other companies,” both vendors and end user firms, he says. The OpenSSF launched an end users working group earlier this year.

End users “build a lot of this kind of internal tooling, and they’re very eager to modernize and update, and align with where industry is heading,” notes Behlendorf.

Protecting the internet’s dark matter

The OpenSSF is also working on memory-safe architectures through Prossimo initiative, which works with Rust and the Linux kernel, and recodes resources such as DNS libraries and even network time in a memory-safe way.

“This is like the dark matter what keeps the internet working. And if that stuff goes out, we’re all in really bad, bad shape,” Behlendorf warns.

OpenSSF will also continue to promote SBOMs as a way to increase supply chain transparency. Behlendorf wants SBOMs to be as universal as readme files within software releases, but concedes this must be done without overburdening upstream developers.

He is also realistic about the prospects for improving open source security, envisaging a future not without vulnerabilities, but where only the “incredibly niche” bugs remain.

“There’s no such thing is defect-free code,” he says. “One of my favorite sayings is that the last bug is fixed when the last user is dead!

“We’ll never get to a point where there’s zero CVEs, but we should get to the point where getting them fixed is a routine part of IT infrastructure upkeep, and is non-disruptive, particularly for critical infrastructure.

“There is that that phrase: ‘with enough eyeballs all bugs are shallow’. There just aren’t enough eyeballs per line of code out there,” Behlendorf concludes.

RECOMMENDED ‘Security teams often fight against developers taking control’ of AppSec: Tanya Janca on the drive to DevSecOps adoption