The Infrastructure Security Engineer Is a Unicorn Among Thoroughbreds

A team at a recent cloud-native industry event laughed out loud when they told us, “We just got out of a talk, and apparently we are now infrastructure security engineers.” With the rampant layoffs in the tech industry, underlying the amusement of job titles is real uncertainty around expectations for thriving in that new role and associated ecosystem.

In the age of Kubernetes and cloud native application deployment, the infrastructure security engineer is a prize hire. But across dozens of job descriptions and practitioner interviews, we found that this role reflects an exceedingly difficult challenge: to be the best at both indirect influence and hard technical skills.

So what is infrastructure security engineering, anyway? The infrastructure or cloud security team sits at (no surprise) the infrastructure layer, versus the application layer. They are primarily concerned with deployment and the running cloud environment.

The first thing to understand about this role is how much the cloud security shared responsibility model requires of them. In the case of managed Kubernetes platforms, we can assume a general platform-as-a-service (PaaS) model. This implies a shared responsibility model that puts nearly the entire configuration of the cloud in the infrastructure security role’s hands. In Google’s own words, “For [Google Kubernetes Engine], you’re responsible for protecting your worker nodes, including deploying patches to the OS, runtime and Kubernetes components, and of course securing your own workload.”

But the shared responsibility model is just the start. No role exists in a vacuum, and the third most common requirement in this role, apart from vulnerability management and staying up to date on trends in the space, is imbuing best practices across other teams in the org. As one hiring manager put it, “Your primary responsibility will be to ensure that our engineering teams integrate security best practices into their workflows and deliver secure products and services.”

There is an inherent friction in asking a development team to do anything that might slow down the flow of new features into production, even if it has been shown that teams baking security into their DevOps processes actually do deliver more quickly.

What Infrastructure Security Engineers Need to Succeed

What do hiring managers think will make candidates successful in the kind of role just described? Not surprisingly, the third most common requirement for this role — behind hands-on experience with cloud platforms and networking — is proficiency in scripting languages, combined with hands-on experience around any combination of infrastructure-as-code (IaC), HashiCorp’s Terraform, and the continuous integration/continuous delivery (CI/CD) pipeline. Why? Because if you have never automated deployments with code, it will be impossible to share security best practices with the developers doing it on a daily basis.

The last common requirement in an infrastructure security role is an in-depth understanding of the end-to-end development pipeline. If a security engineer expects to keep up to speed on the latest in the cloud, influence development, and manage cloud vulnerabilities on a day-to-day basis, they need an understanding of efficiency, how it all works together, and how to prioritize.

Here are some more tips from our interviewees:

• “If you are just looking at the cloud, don’t forget Kubernetes. While it is deployed through managed cloud services more often than not these days, it cannot be addressed in the same way one would address vulnerabilities for cloud environments.” — Director of cloud security
• “Triage is critical. When my teams have failed in the past, it was usually because we kept chasing shiny things. By being disciplined and methodical about prioritization, we maintain confidence that we’re working on the right problems at [almost] any given time.” — Manager, infrastructure and IT security
• “Don’t underestimate engineering teams’ interest in solving security problems. Empower them with data and context, and see how hungry they are to use it.” — Manager, infrastructure and IT security

Why This Could Be the Hardest Job

Interestingly, in our research only one job description had a line item for “security reviews,” where the role allowed the security team to say yes or no to development changes. This is telling in the context of other observations on the role of direct versus indirect influence over engineering and development; for example, the IaC knowledge is needed not for using it directly, but for being able to tell others how to use it.

Also, communication and mentoring were not listed among the most common job prerequisites, but half of the roles still had high expectations for this soft skill. This was especially true for the more senior positions.

Between the requirement to influence the development teams, the required knowledge of IaC tooling and automation, the need for communication and mentoring, and the near complete absence of formal security reviews, a view of the most successful infrastructure security professional begins to emerge. This person will have broad hands-on experience in the cloud ecosystem, as well as skills to influence and build credibility across skilled teams who are managing highly new, cutting-edge GitOps tools on a daily basis. That is a high bar, indeed!