Increasing trust, commitment, and predictability during a remote incident response

Tuesday, January 10, 2023 12:01

Authors: Gergana Karadzhova, Joe Schumacher, Pawel Bosek

In this blog post, Cisco Talos Incident Response (Talos IR) presents some of the key benefits of remote IR support and offers a list of recommendations for working on a remote incident.

Some organizations see added value in having incident responders on site during an emergency. While this approach may offer certain benefits in terms of coordination, in Talos IR’s experience, the physical presence of a team on site is not crucial for the success of the overall IR. Cybersecurity threats are by definition intangible and bringing people physically together does not automatically facilitate an investigation. The traces of the malicious actors involved in a ransomware case, for example, are in the cyberspace made up by an organization’s network and the Internet, which means that, with sufficient connectivity, a remote responder team can work on the case from anywhere.

As a remote-first, follow-the-sun global team, Talos IR has extensive experience in creating a healthy, effective, and collaborative environment for customers and responders regardless of the stressful nature of IR activities. Trust needs to be built over time and proactive IR services offer the means to do this preemptively, ensuring that when an emergency happens, the responders will be able to gain appropriate access in a timely manner. Specific recommendations on how to strategically use proactive services in building this relationship are outlined in section “Adopting a ‘trust but verify’ approach.”

Analyzing the advantages of remote IR support

Criteria

Remote

On-Site

Start of engagement

15 minutes to four hours

24 hours in transit

Personnel utilization and treatment

“Follow the sun” model, ensuring that team members rest in between shifts and lower the risk of burnout

Usually limited to daytime.

Risk of overstretching the team if working around the clock for multiple days

Scalability

Limited only by the IR team’s size

1 – 2 people depending on the size of retainer

Knowledge transfer between IR team and internal teams

Primarily written

Primarily verbal

Efficiency

Overall shorter period of time to start; easier to ramp up; better personnel utilization and care; transparent and lower cost

Overall longer time to start; harder to ramp up; limited daily period of work; additional costs due to travel and accommodation

Remote incident response offers the following key advantages in comparison to on-site support:

• Faster initial response – The average industry service-level agreement for having on-site support is 24 hours in transit. Depending on the exact time when the incident was reported, the transportation options to the location, and the immediate availability of the consultants going on site, it can take anywhere between eight to 24 hours to reach the customer premises. This is significantly longer than the time needed to kick off a remote response, which in Talos IR’s experience is less than two hours.
• Easier to scale the response team – IR is a team sport. Customers usually mainly interact with a few dedicated team members of the IR team, such as an Incident Commander and a few lead consultants. In the background, however, there are additional technology specialists, project managers and threat intelligence analysts working on the case. It is much easier to involve additional personnel in a remotely managed incident than to order people on site.
• More resource efficient – The three most valuable resources in an incident are time, budget, and human power. In terms of human power, consider an incident which has been active for 48+ hours and is still ongoing. In such cases, having a remote team consisting of members in different time zones can ensure that work on the incident follows the sun without burning the team out. When it comes to cost management, remote incident support tends to provide great transparency of activities performed by the team, due to the centralized flow of communication and information exchange. Additional cost efficiency is generated by the fact that technical personnel do not need to set up their workplace in a new environment with emergency conditions and thus suffer a dip in productivity.

Preparing for remote incident response

From inception, Talos IR has been handling our emergency IR while fully remote as the norm, even predating the COVID-19 pandemic, due to the 24x7 nature of our service and global team distribution. Webex organically allows Talos IR to quickly and securely support our customers anywhere in the world. During the pandemic, we supported Cisco customers without any service interruption and created additional virtual infrastructure to ensure that all Talos IR proactive services could be securely delivered. This being said, if we do encounter a special case which might necessitate on-site presence of the response team, we discuss with the customer if it would augment the overall IR effort.

Remote IR comes with its own set of challenges. Talos IR recommends discussing your approach to the points listed below before an actual emergency, as addressing shortcomings might require lead time and resources.

• Provisioning Access – Having a process to get the remote IR team access to on-premises and remote consoles will accelerate the ability to investigate the incident. This process should include accounts related to, but not necessarily limited to, Virtual Private Network (VPN), Active Directory (AD), Endpoint Detection and Response (EDR), Multi-Factor Authentication (MFA), and other security controls.
• Regulatory & Compliance Requirements – Data collection and sharing are often subject to country-specific laws, especially in the case of critical infrastructure and personally identifiable information (PII). Technical teams should work with the Legal Department and the Data Privacy and Governance Office to verify the exact legislation that is applicable to the different data sources that might be part of an incident. Any known limitations regarding remote collection of evidence and analysis of the data should be documented in the Incident Response Plan.
• Communication Cadence – Having multiple means of communication and a set status update meeting will help ensure that everyone is kept on the same page. Communication is critical to get right during an incident but does not have to be in-person to be efficient or effective. People tend to be hyper-connected during IR, so bringing structure and predictability to the information flow benefits everyone.
• Gathering Evidence – Remote evidence collection might not be possible in all incidents, but with some planning can be a viable alternative to traveling to a location or shipping an asset. You will want to ensure that there are technical representatives on-site who can assist the IR team with collecting forensic and triage data from in-scope assets.
• Sharing Evidence – A reliable, high-speed network connection is essential for downloading and uploading large files to remote server(s). You will most likely be downloading the requested evidence from endpoints (e.g., workstations and servers) within your internal network and then uploading this data to a remote server for the incident responders to analyze.
• Scaling the Response – Typically, an incident starts with one or two events across a large, networked environment and expands in scope as more indicators of compromise are discovered. Having the ability to contact people across the team as the incident evolves and to connect them to the investigation remotely will reduce the overall time to act and contribute to a more efficient incident response. When planning who and how you might need to engage during a possible incident, it is critical to consider your internal team along with the partners, contractors, and service providers that will be assisting with the incident response remotely.

Internal teams that will be involved in the implementation of the above preparation activities should familiarize themselves with the agreed approach as part of their preparation for IR and whenever possible, test the theoretical plan in the practice. It is highly recommended to document all relevant procedures and processes within the internal Incident Response Plan and Incident Response Playbooks, as this makes distribution and knowledge management easier in the long run.

Adopting a “trust but verify” approach

In the world of security there is the phrase “trust but verify,” and this mentality should be present when it comes to your IR service provider. Many companies have their first contact with the IR team during an emergency, which is one of the worst times for building trust and getting to know each other. A much better way to become familiar with your incident responders is to work with them on proactive services. These services by definition are non-emergency and offer more time for addressing the remote support considerations listed above through tabletop exercises, development of plans and playbooks, and threat hunts. In addition to improving your overall resiliency to cyber threats, proactive services will allow you to see what tools are typically leveraged by the IR team and what methodology they use for threat hunting. The engagements provide an opportunity for the responders and your internal teams to build trust and a better understanding of each other’s craft. An investment in the human element of virtual collaboration pays off during subsequent remote IR cases, as both sides feel more comfortable working together and can be more efficient in the execution of emergency response actions.

In most businesses, a level of remoteness was present long before the COVID-19 pandemic. Many organizations operate remotely or in different locations, whether that is multiple offices, data centers, or the homes of their employees. The increase of remote work in the last few years has led to optimization and normalization of virtual collaboration, the latter becoming an integral part of modern business operations. In a similar way, IR has grown increasingly independent of physical location as long as secure connectivity is possible. There will always be the need to physically travel to a location for response in some special cases, but that will not be the norm.

The expanded use of cloud services, the increase in remoteness in company operations, and the shortage of cybersecurity talent all speak in favor of an IR delivery model with remote-first support and members distributed around the globe. This trend emphasizes the importance of accountability and trustworthiness in each and every interaction with the IR team, whether physically or virtually. Talos IR recognizes this and is committed to delivering high-quality and efficient engagements during all emergency and proactive retainer projects.