Proactive Threat Hunting Case Study: GhostCat

Proactive Threat Hunting Case Study: GhostCat

The active exploitation of a vulnerability known as ‘GhostCat’ was discovered when a VMware Carbon Black Threat Analyst conducted a proactive, post-mortem threat hunt within a previously compromised customer’s environment.

The Original Attack

In early 2022, the VMware Carbon Black Managed Detection & Response team notified a customer when Apache Tomcat was observed invoking werfault.exe, indicating the attempted exploitation of the Log4Shell vulnerability (CVE-2021-44228).

Figure 1: Initial Log4Shell alert.

The endpoint in question did not have a VMware Carbon Black sensor installed when it was initially compromised, and although the customer was observing and attempting to remove leftover artifacts of the compromise such as malicious executables and cryotominers, the root cause had yet to be determined. Determining root cause can be a daunting task especially when attackers use multiple techniques for obfuscation and diversion to maintain persistence within the target device. Once our sensor was installed on the device, however, it didn’t take long to see that the true problem was an attacker exploiting the infamous Log4j vulnerability that had gone unpatched in the environment due to lack of visibility.  This allowed the attacker to execute fileless encoded powershell commands which attempted to pull down malicious executables hosted remotely.

Figure 2: Decoded powershell command (defanged).

VMware Carbon Black Threat Analysts detected and emailed on the observed malicious activities immediately and gave the customer the information they needed to easily remediate the underlying root cause issue.  We were able to provide timely and personalized policy and containment recommendations that identified numerous unwanted network connections and previously undetected artifacts.  This became one of the very first cases of Log4Shell exploitation we found and effectively responded to in the wild, but it would certainly not be the last.

GhostCat Discovery

When unique or significant threats like the one previously described are found in customer environments, our team documents these attacks for improved detections, post-incident threat hunting, and future training.  Preserving the indicators of attacks affords the Managed Detection and Response team the ability to continue to monitor threats as their Tactics, Techniques and Procedures evolve. During a post-mortem review of this particular incident, a Threat Analyst conducted a proactive threat hunt within the aforementioned customer’s environment - specifically to ensure that all activity remained blocked and to test several new possible Log4Shell queries and policy modifications to better prevent future malicious activity.

While conducting this exercise, Tomcat was seen regularly accepting TCP/8009 connections from various suspicious international IPs.  The source IPs for these connections originated from all over the world, including many international conflict hotspots.  It is common that attackers use VPNs to route their traffic through different locations than their own in order to remain anonymous, so this was the first indication that some malicious activity may still be occurring in the environment.

Figure 3: International network connections to Tomcat over port 8009.

Port 8009, along with 8005 and 8080, is open by default for Apache Tomcat, and while users often close port 8080 to block external HTTP traffic, 8009 is commonly forgotten about.  If left exposed to the public, it can provide similar access to internal components.  Periodic connections to Apache Tomcat servers over this port, and the nefarious nature of some of the IPs connecting to this device, encouraged the analyst to bring these findings to the rest of the team to dig deeper.  The team collectively searched for new indicators and found signs that pointed towards the active exploitation of a known Tomcat vulnerability (CVE-2020-1938), commonly known as GhostCat.

Potential Impact of GhostCat

Ghostcat is a vulnerability that affects the Apache JServ Protocol (AJP).  AJP is responsible for communication between a webserver and an Apache Tomcat Server.  Most of the time when GhostCat materializes, it does so as a Local File Inclusion (LFI) vulnerability but occasionally even allows for Remote Code Execution (RCE) if there are significant misconfigurations or additional tampering. Due to the complexities of the requirements for GhostCat to produce an RCE impact, it is highly unlikely to happen in the wild.  The less severe, yet much more common, LFI impact still allows an attacker to abuse proxy requests to Tomcat port 8009 in order to obtain sensitive information such as configuration files and even source code of the deployed applications.  Essentially, the possibilities range from information leakage to arbitrary code execution using internal protocols.

Shared Successes

To fix the vulnerability, we advised the customer to update to the latest version of TomCat, which contains a patch for this vulnerability.  Additionally, we gave alternative fixes in case the customer was not able to perform this update for whatever reason.  We also provided the customer with article recommendations to read further about this vulnerability, links to information about the patch for it, and briefly revisited their policies for additional recommendations to stop this sort of activity in the future. 

For VMware Carbon Black, simply detecting malicious activity and notifying a customer is not enough.  We are constantly updating detections, investing in the education of our Threat Analysts on new threats, and ensuring our customers have everything they need to not only remediate an incident but remain safe and free to focus on their business objectives.  Proactive threat hunting is just one example of how customers not only get the service they pay for but  also benefit from the passion our analysts show for their jobs and the intelligence we gain from working in hundreds of unique environments.  Each customer brings their own unique potential for vulnerabilities and peripheral considerations to the table, and it is our task to keep this in mind when assessing an incident.  Each attack we see is handled, documented if it is new, and meticulously dissected in order to increase our potential to recognize the warning signs the next time we see it.

Proactive threat hunts allow our Threat Analysts to search for, and dive deep into new vulnerabilities they have never seen before, such as GhostCat.  For our customers, this internally conducted research goes unseen, but is massively beneficial.  By making custom queries that look for very specific activity, we are able to comb through event logs and potentially find artifacts that have been overlooked by the customers’ internal security teams.

Our Threat Analysts prioritize high severity alerts in customer environments and will manually threat hunt for additional suspicious activity.  In a threat hunt, we evaluate normal events that haven’t triggered alerts looking for opportunities to improve our detection capabilities.  Additionally, we find low severity alerts and reassess the ranking associated with those detections.  In this instance, the customer had been dismissing the activity as normal behavior for their Apache Tomcat servers, but after our analysts notified them of the highly unusual behavior, they were immediately able to stop the international connections.  These connections have not been seen since in the organization, confirming the suspicion that it was not intentional.

Conclusion

When you partner with VMware Carbon Black, you aren’t just subscribing to leading detection capabilities, you are also partnering with our Threat Analysts, who are passionate about what they do, eager to find threats, and intent on helping improve our customers’ security postures.  This case is a prime example of how our post-mortem analysis process provides value for everyone involved; the customer was notified of malicious behavior twice, and we were able to document an attack we had not previously observed before. Having this knowledge is crucial, because while VMware’s Threat Analysis Unit (TAU) is always identifying new threats and creating detections for all Carbon Black customers to benefit, Managed Detection and Response customers are additionally positively impacted when our Threat Analysts increase the range of unique threats they are able to quickly identify.