Open source incident response solutions

Red Hat leads the tech industry’s cutting edge practices for the resolution of cybersecurity issues. Red Hat does this by providing relevant and accessible information and enabling the larger community to make well-informed decisions about security issues.

As part of our continuing reviews, Red Hat saw the need to make public a formal incident response plan (IRP) to lead our incident response and vulnerability management. FedRAMP and other regulatory frameworks also require a formal, published IRP. It made sense that Red Hat should put forth the effort to make sure we thoroughly documented our incident response processes to cover our needs and to deliver a more systematic way to analyze and improve our vulnerability reports.

As we researched how other companies handled the reporting of vulnerabilities, we quickly discovered that there are no open source IRPs for product security. Since Red Hat fosters a culture of innovation, we decided to formalize our own IRP and make it public.

We also decided that we would live true to our open source ethos and obtain feedback from the community. As a result, we have published a template for industry use and consideration. This document is the first public, open source Product Security Incident Response Plan created, and we look forward to collaborating with industry partners to improve our security processes.

Why have an IRP?

An incident response plan is a planned course of action for all significant security incidents. Some incidents lead to larger efforts impacting products for days or months. Having an incident response plan helps stop, contain, communicate and resolve incidents more quickly in an efficient manner with greater consistency. It is not a playbook, but rather an overarching guide to the processes that need to happen across the organization around incidents and their resolution. After all, incident responses involve more than just the security team and engineering. Playbooks and other detailed procedures are then linked to the plan.

Another example of the value of an IRP is how it informs and collaborates on specific processes that support the response effort for the organization. Red Hat includes how to classify the severity of each Common Vulnerability and Exposure (CVE), additionally providing a Common Vulnerability Scoring System (CVSS) score. However, a particular Red Hat product may be less impacted due to compensating controls around a specific piece of code. A formal IRP helps direct the teams in responding to these scenarios and gives a solid response to customer questions about being thorough in responding.

Our process

Having a systematic process that can handle these requirements is not trivial. Red Hat’s plan for incident response is a multistep process that starts by triaging flaws, then doing analysis and finally following through with trackers and fixes. The IRP is the formal process that Red Hat will follow when presented with a product security incident. These incidents can be as simple as a false positive report or a severe risk to the security of our customers using our products. When we receive a flaw, the first step is to determine if our products are affected. If so, we determine the severity to which the product is affected. This severity analysis will determine the urgency of the response and ensure that the vulnerabilities are fixed promptly. This process must be followed consistently and accurately.

For companies, like Red Hat, that are involved in numerous open source communities and vulnerability email lists, the first step, “triage,” can be challenging. These disclosure lists publish all known vulnerabilities and place the responsibility on a company to determine the effect on their products. In addition to reviewing these sources, we maintain an email address where people can report vulnerabilities. All of these information sources form the basis of our reports, which we monitor in a queue system. When an incident is reported in this queue system, this kicks off our assessment for severity and notification.

As a result, while many vulnerabilities do not affect us, we must triage them to determine whether we are affected or not with certainty. This “lack of guaranteed risk” presents numerous challenges to our analysis process.

If it seems plausible that our products may be affected, we send the vulnerability for further analysis, entering our assessment and coordination phase. This is where we do a complete analysis. We verify whether we are affected by the vulnerability, and if we are, we coordinate with engineering to ensure a fix is released promptly.

Some CVEs are not public when reported via direct private communications or private mailing lists, known as Embargoed CVEs. In these cases, there must be a way to keep them private while coordinating the fix and until they are disclosed to the public.

This IRP process helps to protect customers as soon as reasonably possible for various flaws. When we have a decision to create a fix and a timeline for a fix to be ready, we conclude this phase and enter our recovery and closure phase. Here, we finalize the incident’s tracking and prepare any necessary outside communications. This concludes our final phase, which finishes our consistent analysis process.

Our IRP document tracks the primary stakeholders we interact with and our expectations on what they will be tasked with during incidents. For example, we will work with Engineering, Quality Engineering and Release teams to track reported incidents impacting each product engineering team through the release and closure or decision not to release a fix. When an incident is classed as a Major Incident, our process guides us on when we will engage with each stakeholder for internal tracking and orchestration. This engagement during a Major Incident extends to valuable contributions from Legal and Communications teams for preparing public communications for our customers and other factors outlined within the IRP.

Every phase in our process has a clear list of steps that must have a conclusion, prevent missed details and function as a requirements checklist. As a result of this orderly method, the analysis is much faster and analysts can cover more products, adding to more accuracy.

Conclusion

We believe that sharing this methodology with the broader software community helps provide us all with more secure software as well as coordinated orchestration for vulnerability responses. This methodology continues to enable us to be more proactive in software security, where our response is planned and understood by all stakeholders, making it faster and more consistent.

We also welcome industry partners to collaborate with us on this so that we can all collectively improve our product security incident responses.