Where Are All of the Container Breaches?

How will threat actors attack and utilize containers? This is a question I constantly think about. I’ve been working in this area for more than two decades now, and I feel like I should have an answer. But I don’t.

Instead, I have a lot of different ideas, none of which I can really pinpoint as correct. Part of this indecision is because of all the time I spent learning security in the “legacy” world. Containers do not really have an analog. Sure, virtual machines (VMs) are often conflated with containers, but they were never able to scale like containers. They are also used for completely different purposes than containers. It’s taken a while to adjust my thinking and understand where containers actually fit into the attack surface.

The public examples of attacks against containerized environments are very limited. The breaches almost always are cryptomining related, which are serious attacks, but the incident responder in me finds them underwhelming. Another commonality is they are mostly the result of a misconfiguration, whether that be in Kubernetes or a cloud account. The combination of the motives and the tactics have not been very inspiring so far.

The Old Way

Remote code execution (RCE) vulnerabilities have been the main concern in computer security for a long time. They are still, but how does this way of thinking apply to containers? It’s easy to immediately jump to RCE as the primary threat, but it doesn’t seem to be the right way to approach containers. For one thing, containers are often very short lived – 44% of containers live less than five minutes – so an intruder would have to be quick.

This approach also assumes that the container is exposed to the Internet. Certainly some containers are set up this way, but they often are very simple and use well-tested technologies, like NGINX. There may be zero-days out for these applications, but they would be extremely valuable and hard to come by. My experience has shown me that a lot of containers are used internally and don’t connect directly to the Internet. The RCE scenario gets a lot more difficult in that case. I should mention Log4j, though these sorts of vulnerabilities have the potential of being exploited remotely even if the vulnerable system isn’t on the edge.

The New Way

If RCE isn’t the biggest threat facing containers, then what is? Are containers even on the radar of threat actors? Yes, containers and their supporting infrastructure are too important to ignore. Container orchestration software has allowed containerized workloads to be scaled to unimaginable numbers. The usage trend is also increasing, so you can be sure they will be a target. They just can’t be thought of like servers you get at through RCE vulnerabilities.

Instead, the reverse is actually true. Instead of attacking containers from the outside in, they need to be attacked from the inside out. This is essentially what supply chain attacks do. Supply chain is an extremely effective attack vector against containers when you start to understand how they are built. A container starts with a definition file, such as Dockerfile, which defines everything that will be in the container when it runs. It is turned into an image once built, and that image is what may be spun up into a workload an innumerable amount of times. If anything in that definition file is compromised, then every single workload that runs is compromised.

Containers are often, but not always, purpose-built with an application that does something and exits. These applications can be almost anything — the important thing to understand is how much is built using libraries, either closed source or open source, written by other people. GitHub has millions of projects, and that’s not the only repository of code out there. As we saw with SolarWinds, closed source is vulnerable to supply chain attacks as well.

A supply chain attack is a great way for threat actors to get into a target’s container environment. They can even let the customer’s infrastructure scale their attack for them if the compromise goes unnoticed. This type of scenario is already playing itself out, as we saw with the Codecov breach. But it is difficult to detect due to how new all of this is and how our thinking is still rooted in the problems of the past.

A Way Forward

As with fixing most problems, visibility is usually a great place to start. It’s hard to fix what you can’t see. To secure your containers you need to have visibility into the containers themselves, as well as the whole pipeline that builds them. Vulnerability management is one type of visibility that must be integrated into the build pipeline. I would also include other static analysis tools, such as ones that look for leaked secrets to that as well. Since what a supply chain attack looks like can’t really be predicted, runtime monitoring becomes critical so you know exactly what your containers are doing.