Surviving a Major Cyber Incident
The first 72 hours of response
Effective cyber security is a balancing act between the availability of skilled people, tested and proven processes, and implementation and gainful use of security technology; with the fulcrum upon which these equal, tripartite elements balance being planning. There is no more real a stress-test that will show just how well it all comes together than how the people, the processes, and the chosen technology come together in a planned fashion during the first, critical 72 hours following a cyber breach.
To give you a deep insight into how the FoxCERT team deals with a cyber incident in the real world, we will take you along for the bird’s eye overview of the first 72 hours of an incident response engagement. It is based on a real incident and gives a good overview of our average Incident Response (IR) engagement. Of course, the details have been altered to preserve the anonymity of our client.
The critical first 72 hours
When a major incident of any kind occurs, including a cyber incident, how the team responds in the first 72 hours has been demonstrated to be the critical factor in terms of the scope of direct and collateral damage. In these first 72 hours the incident response team’s availability, skills, preparedness, and tooling come together in a way that will make a measurable difference to business outcomes.
Before we set the clock ticking, there are two aspects of Fox-IT’s preparations to understand:
Firstly, in early October of 2022 Fox-IT released Dissect1, our incident response framework, as open-source software. Dissect gives the FoxCERT team the proven, repeatable set of processes to follow which means our incident response team knows exactly what works, how decisions are made, and a framework that eliminates much wasted time.
Secondly, for tooling Fox-IT uses VMware Carbon Black as our preferred Endpoint Detection & Response (EDR) solution. Carbon Black EDR is a proven enterprise class with rapid deployment and demonstrated scalability. We also use Splunk as a preferred data aggregation and search platform.
Let’s go.
Hour 0
Friday 18.00
The incident handler on call receives a heads-up from the account management team that a potentially large incident is incoming. 15 minutes later, after rushing home, the incident handler receives a call from the client on the emergency response hotline. The client was notified by law enforcement that they had a security issue. After performing an initial triage by phone, it was confirmed we were dealing with a significant incident. We agree with the client on an initial budget and establish an investigation team. Simultaneously, the client uploads the first pieces of investigation data to the Fox-IT forensic lab.
Friday 21:00
A kickoff with our team (an incident handler and two analysts) and the team of the client takes place. Our team provides initial feedback on the uploaded data and presents a strategy on how to approach the incident. The approach is to get Carbon Black EDR coverage on all systems for live monitoring, and to perform data acquisition on several systems in order to analyze historical events. An addition to FoxCERT’s use of Carbon Black as our preferred EDR platform, for data acquisition we use the in-house developed Acquire which is part of the Dissect framework.
Our data acquisition technique boils down to “give us everything”. We prioritize the acquisition of critical systems (both in importance and likelihood of compromise), but beyond that we want to get as much host data as possible. This allows us to do analysis in bulk, instead of having a back-and-forth with the client each time we want to collect a new system. “Everything” is usually still a multi-step process though, and this time was no exception!
Hour + 5
Friday 23:00
First Carbon Black EDR agent online.
Hour + 7.5
Saturday 01:30. H + 7.5
After ensuring that the continued rollout of Carbon Black EDR is working smoothly, and that the client team knows how to perform data acquisition and upload that data, the initial IR team members take downtime. We agree to provide the client with the next update on Saturday at 14.00 CET, which is early morning for the client. We agree to resume work around 10:00 CET in the morning to have some findings ready by then.
Hour + 15.5
Saturday 09:30. H + 15.5
Thirty minutes ahead of schedule the investigation team is back online and using Carbon Black EDR insights, discovers concerning live activity of the adversary. Based on this finding a decision is made to wake up the client (note: in the middle of the night for the client) to prepare for a meeting in 45 minutes. While it generally feels positive when our approach works, telling a client that they likely need to take important and high impact decisions such as shutting down internet connections and thus operations, can feel uncomfortable. Nonetheless it is important we act based on facts and the evidence currently available to us.
Hour + 16.5
Saturday 10:30
We give a summary presentation to the client and our initial recommendations. As always this includes a discussion regarding several scenarios and their potential business impact and effectiveness for containment. Our client agrees to the incident response teams recommendation regarding a specific containment scenario to cut off internet access[1].
Hour + 20
Saturday 14:00
We have a “start of day” meeting with the client and discuss the approach for today[2]. The goal is to do analysis and get a view on initial foothold, scope of compromise and a timeline.
Hour + 25
Saturday 19:00
We present the client with an update on findings, including an initial timeline and scope of compromise. In addition, discussions are ongoing on how to bring back some of the clients’ core processes in a safe way for the start of the business week.
Hour + 30
Sunday 00:00
After a further update with the client to share additional findings, the FoxCERT team gets some sleep.
Hour + 40
Sunday 10:00
FoxCERT management initiates the process of getting a team together that can provide on-site containment and remediation support for the client. This involves numerous communications between the initial FoxCERT incident response team, and the Remediation Management and Governance (RM&G) group. The IR team and RM&G perform practice drills to ensure smooth execution when on site. Later that morning there is confirmation that a containment lead and a pentester will fly to the customer’s premises the next day.
Hour + 44
Sunday 14:00
“Start of day” meeting including discussion of our approach for today. A new potential backdoor into the network is reported by the investigation team[1]
Hour + 48
Sunday 19:00
Findings meeting with the client. The timeline is expanding. While the meeting is ongoing the team finds evidence regarding the adversary and initial foothold. Decision is made to share the news later that night when there are additional details and observations made regarding the behavior of the attacker, also called an adversary profile.
Hour + 53
Sunday 23:00
End of day meeting. We share attribution information about the identity of the adversary. The adversary’s Tools, Techniques and Procedures (TTP’s) we observed match with content in several blogs and allow us to provide attribution with a “moderate confidence” rating. Information on potential goals and TTP of the adversary is used by the client and the investigation team to improve decision making on how to further mitigate the attack. It turns out that cutting the internet off was probably a very good idea and was taken at just the right time.
Hour + 68
Monday 14:00
“Start of day” meeting. Each member of the team worked at least 30 hours over the weekend. The team discusses the approach for today and agrees with the client on a bit shorter working day that day.
Hour + 72
Monday 18:00
We develop a more complete picture of the incident. Based on this we conclude that the timeline of the attack likely extends significantly (multiple months) into the past. In other words, the attacker gained an initial foothold into the client’s network months prior to them being tipped off by law enforcement (which was the trigger for them calling FoxCERT and for the clock starting on these 72 hours). Throughout those months the attacker had remained undetected.
Our incident response team also confirms the initial intrusion vector. Meanwhile, the on-site Fox containment team arrived on-site and started in their continuous support of the client in advising about mitigation and containment and even how to re-start some business processes, while doing this in a secure way.
This secure way is devised by looking at the techniques used by the attacker which were found by the investigation team and making sure that this process is not sabotaged or disturbed by the attacker.
Wrapping up
Combining different artefact sources for triaging purposes is nothing new when we’re talking about incident response. With Dissect we were able to provide the client with a picture of the incident using the data sources of ~100 systems and only two host investigators within 72 hours. During the remainder of the incident this number reached around ~250 systems, but still only two host investigators were involved.
During the first 72 hours of incident response good decision making is informed by accurate data. The FoxCERT team relies on the combination of the Carbon Black EDR agent to gives us the ability to monitor the environment in real-time, together with the Dissect framework and Splunk to analyze historical artefacts. Both sides feeding each other with indicators and more insights into the environment and the attack.
With this information we could quickly and aptly answer the questions the client had regarding the scope of the compromise, create a complete timeline of the path the attacker took within the network, as well as all the pivots and toolsets that were used, and very importantly to visually show this to the client.
We hope you enjoyed reading this short war story where we give you some insights in how we at Fox-IT leverage Dissect in our incident response practice to help our clients.
Your 72 hour countdown could start at any time. Will you be ready?
Associated Content
From the action bar MORE button.