Understanding open source software supply chain risks

How many people does it take to buy a pair of shoes? It’s kind of a trick question. In a literal sense, it takes you and the person you are interacting with (assuming you are in a physical store). But the spirit of the question is: How many people are involved in the process of buying a pair of shoes? There’s a salesperson, store manager, shipping and logistics companies, the manufacturer of the shoes, manufacturers of the tools and equipment, manufacturers of raw materials—the list goes on. Each separate entity is part of the supply chain.

The phrase “supply chain” is appropriate because each of those suppliers can be thought of as a link in a chain. You can also start to understand how our global economy and livelihoods are complex and very interconnected. The shoe example can be an interesting thought experiment because virtually everyone is a supplier to someone else (When was the last time you seriously considered whether your aglet supplier was untrustworthy?).

But what if, instead of discussing shoes, we discussed purchasing a software solution? The concept is pretty much the same, and the supply chain size could depend upon the software in question. Even an in-house custom software solution has its own supply chain, but for this blog post, let’s talk about the software you are purchasing, specifically open source software.

Open source community members contribute code to upstream projects, and this is roughly where the open source software supply chain starts. Many organizations and individuals contribute to this code from all over the world for free. From there, companies and other entities take the code and distribute and support it for a fee or incorporate it into a product they sell.

For example, companies like Red Hat sell subscriptions for support and ongoing maintenance so that our customers don’t have to do it for themselves (which is certainly an option). Other companies use open source software in a product they manufacture, like IoT devices. It’s likely you already own a device that is running open source software today.

Based on these examples, you may start to visualize the software supply chain. By the way, if you want to check if a device is using open source software, a notice is usually placed somewhere within the user interface (e.g., under a “help” or “support” menu option).

Learn more about the software supply chain in this episode of Security Detail.

**What is a supply chain attack? **

Attacking a supply chain is not a new thing; this tactic has been around for a long time. For example, as part of a military conflict, an adversary may attempt to disrupt or destroy their enemy’s supply chain (like food or artillery) or to gain a tactical or strategic advantage. However, a software supply chain attack is very different from the example above and somewhat different from other types of cyber attacks. But, in all these cases, (and at a high level) the attacker is looking for a weakness in the code to exploit. Software supply chain attacks are typically (but not necessarily always) used as a vehicle to gain access to something else. In other words, the “maker” is not the target; they are just a means to an end.

Early examples of supply chain attacks made headlines when large retailers and home improvement stores experienced breaches due to a weakness in their supply chains. A software vendor was breached to gain access to the retailer’s internal network. In this case and others like it, the attackers were after data—specifically, their customers’ credit card and other sensitive information. Unfortunately, that breach led to other issues that made the attacker’s job even easier. I’ll touch on that a bit more in a minute. More recently, software supply chain attacks have become more sophisticated as cyber criminals learn and adapt to improving security standards and practices.

Another supply chain attack made global headlines in early 2020. The attackers gained access to the company’s source code repositories and inserted malicious code, which, in turn, exposed the company’s customer base. Again, the software solution provider was not the end goal; it was their customers’ data—presumably for espionage and other criminal activity. In this particular case, “state-sponsored” actors are suspected. And in yet another case, a supply chain was severely disrupted when an energy company’s network was compromised as part of a ransomware attack. This attack had an immediate physical and economic impact. I point this particular incident out because the ramifications went beyond the digital realm and resulted in real world consequences. Where I live, a cyber attack literally led to long lines at the gas pumps. We can and should expect future supply chain attacks to get more sophisticated.

Why is this important, and why should you care?

If my examples above weren’t enough – As my colleague, Alison Naylor, mentioned in her blog post about IoT devices and their security, our world is becoming increasingly connected—we have connected devices virtually everywhere. All of those devices use some form of software to run them (also called firmware). It would be challenging, if even possible, to go about your daily routine without interacting with something that runs some form of software. Your home, car, local grocery store, that gas station around the corner (and the list goes on and on) are all using devices that depend upon software. And more likely than not, they are connected to each other. It is imperative that manufacturers of these devices treat their source code appropriately—but let’s come back to that in a moment. The point is that we are virtually surrounded by things running software, and our daily lives depend upon that software to do our jobs or manage our lives. Software suppliers must ensure that their code is as secure and safe as possible.

What can (and should) you do to secure your supply chain?

Given that security is not a destination you arrive at or can declare “victory” over, you should have a security program that continues to evolve, educate and promote security practices. Your information security team should have established policies that enforce and verify compliance. Remember the incident above where I mentioned the attacker’s job was made even easier? It turned out that the company’s established best practices around storing sensitive data were not being followed, allowing the attacker to access the data in the clear. You should also have a product security (or similar) organization whose mission is to validate your software’s integrity. It is no longer acceptable that your team just mitigates the latest vulnerabilities in the code. Your product security organization should partner with your information security organization to drive fundamental security practices on the systems and services that make up your software pipeline. But it doesn’t stop there. Product security should also be looking at and planning to adopt practices such as NIST’s Secure Software Development Framework (SSDF) or Supply-Chain Levels for Software Artifacts (SLSA) if they have not already done so. Not only does following those sorts of frameworks put you on the right track for enhancing the security of your piece of the supply chain, it proves to your customers that you’re doing what you should be doing.

An attempted cyber attack is not a question of if it happens. It will happen. Your job as an associate of your company (regardless of your role) is to help outright prevent or at least limit the success of an attack. Your security program should also plan accordingly, and having a resiliency plan is also key to a solid security program. A disruption to your business could be extremely costly, damaging or worse. There are multitudes of statistics and studies around this, but one statistic that jumped out at me recently was this: “96% of businesses with a disaster recovery plan in place fully recover operations.” That is an astounding success rate!

This post is certainly not meant to be a roadmap for developing a plan to improve the security of your supply chain, but it should give you some things to think about and probable directions to start in. At the end of the day, developing and implementing security plans depends on what is most appropriate for your organization, including the risks you are potentially facing and willing to accept. A large multinational company is more likely to both be at risk of attack and unwilling to accept some risk levels than a local business around the corner would be, for example.

Where to begin (certainly not an exhaustive list):

• Do you have an information security policy or a set of guidelines?
  • Is it enforced? Just having a policy somewhere is only marginally better than not having one at all if it isn’t implemented.
  • Is it regularly audited for compliance?
• Do you have a plan, or are you in the process of adopting an industry standard for handling software and its source code?
• Do you have a resiliency plan?
  • Do you test it?
• Does your organization assess the risk of your vendors? In other words, are you asking your suppliers these same tough questions?
  • You may be thinking, “I/They don’t have a right to know that! It’s none of my/their business!” It could feel a little nosey or even cringy, like you are encroaching on their property.
  • Ah! But it is your business! You have the right to know that your vendor is not potentially the next attack vector to your organization or, worse, your customers. You also want to know that they have the ability to not only mitigate but survive an attack. What would you do if a critical supplier suddenly went out of business because they couldn’t survive a disaster of some sort? What would happen to your business?

Tip: These are all very large, complex and sometimes very expensive problems to solve. Break these problems down into smaller problems. Determining your levels of risk and risk acceptance will help you do this and prioritize where you should focus your efforts first. Trying to solve all these problems at once will likely result in making minuscule progress on multiple fronts, and causing organizational frustration and burnout, with almost no risks averted or mitigated at all.

One last thought for all of us who work in this field: While we don’t want to be the “weakest link” in the chain (either personally or organizationally), that position isn’t good enough. It’s imperative that, as a community of software developers, suppliers and consumers, we hold each other accountable. It is entirely acceptable to ask your suppliers to ensure—and even prove—that they are meeting specific standards, regulations, etc. Your customers may already be asking this of your organization. That sort of accountability only strengthens us as a community.

David Mair started with Red Hat Linux while in college. After graduating from Eastern Kentucky University, he started work with IBM, where his Linux hobby became a job. Shortly after achieving his RHCE, he joined Red Hat in Raleigh, NC, in 2004. While at Red Hat, Mair completed his MS in Information Security at Capitol College, Maryland. He moved into a people management role a short time later and has led different teams at Red Hat. Most recently, he has found a home he couldn’t be happier with in Red Hat’s Product Security organization. He currently leads the Enablement team within the Supply Chain security organization of Product Security.

Read full bio