VMware's MITRE Engenuity ATT&CK® Evaluations delivers Unmatched Out-of-the-Box Security Value

VMware is pleased to announce the results of the fourth round of the MITRE ATT&CK Engenuity Enterprise Evaluations where VMware proved the Day 1 value of the VMware Carbon Black Cloud with NSX Network Detection & Response. Both the detection and prevention tests clearly demonstrate that VMware provides the only solution for security operations centers (SOCs) that delivers maximum security value out-of-the-box against even the most sophisticated adversaries.

Key Highlight: These results were obtained using out-of-the-box policies and default threat intelligence feeds available to any VMware customer, something other vendors charge extra for.

MITRE Engenuity’s rigorous testing reflects the type of threats that our customers are seeing in the real world, where there are no redo’s. With highly effective detection and prevention, our results demonstrate the power of VMware’s comprehensive endpoint and network visibility, which not only works out of the box but also works the first time.

Many other vendors in the MITRE Engenuity ATT&CK Evaluations have resorted to submitting sensors and configurations that are highly engineered toward success on what is essentially an open book evaluation. While this may tick all the boxes for MITRE for an easy A on the test, it is becoming increasingly apparent that such configurations are incredibly false-positive prone and ultimately unusable in a real-world environment. However, since there is no penalty for false positives on MITRE Engenuity ATT&CK Evaluations, most vendors unabashedly use extra aggressive configurations that would only succeed in drowning an analyst in unnecessary alerts.

If you would like to review the configurations of the various participants, navigate to the Participants page on MITRE Engenuity’s official website, and in the top corner select the select link for the most recent round of testing. At least one vendor claiming victory apparently didn’t bother to submit the details of the configuration they used on the test in time for the published results.

Zero Configuration Changes: Why It Matters

In addition to some interesting product configurations, perhaps the best-kept secret of the MITRE Engenuity ATT&CK Evaluation is the notion that at any point during testing, a vendor can say to MITRE, “Hey our product missed that last portion of the test you ran. We’re going to make some changes and then we’d like you to run that part again.” MITRE Engenuity defines this officially as a “Configuration Change.” This is a modifier attached to a detection indicating that a vendor ultimately received more credit for detection than they originally earned. These Configuration Changes can either be a code change on the sensor to collect more or different data, backend changes to the product’s detection logic, or changes to the product’s user interface to surface more data than what was originally shown to MITRE. Here’s how MITRE describes them:
 

This might be the first time you’ve heard about this, as many vendors who constantly utilize Configuration Changes to boost their analytic coverage and visibility scores probably do not want to attract any scrutiny in this area. These Configuration Changes might even seem harmless until you realize that in a real-world scenario a customer cannot make any of these changes themselves. If you’re a customer-facing a cyber incident, every leading solution tested other than VMware would likely require you to submit a customer support ticket to the vendor during an incident so that vendor could write some new code for your sensor, manually tweak the detection logic, or surface more data for you in the UI. Hopefully, they respond to your ticket quickly, but this fact alone amounts to a sizable risk that could add countless hours or days to your response time just to replicate the same visibility that vendors are claiming their results automatically exhibit.

The following graph shows which vendors in Round 4 have been forced to take advantage of the opportunity to repeatedly request a redo from MITRE because their product can’t see all the adversary tactics, techniques, and procedures (TTP) the first time around. Some vendors’ products needed a lot of hand-holding to get it right.

VMware is the only vendor to submit its solution with an out-of-the-box configuration and not make any vendor-specific changes after the start of the test. This is crucial to ensure that our configuration is replicable by customers in a real-world situation. By making dozens of Configuration Changes after the test begins, all the vendor is proving is that they have a team of people working very hard to prop up their product while they know the evaluation is occurring. While there is nothing wrong with an active threat-informed defense exercise, it is unlikely the average customer enjoys that same level of white-glove treatment on a daily basis. We see a lot of “grade inflation” as a result across all vendors and suddenly the true value of the product to an actual customer becomes rather nebulous.

The interesting part of all of this is the vendor knows in advance what adversaries are being tested, when the tests are being run, and they have plenty of time to configure their product to do well. It’s somewhat alarming that even after all this preparation for a planned test, most vendors still needed to reconfigure their offerings five, ten, as many as twenty-five times after the test began in order to be competitive. Results from vendors who make multiple changes demonstrate an inability to properly scale to meet the needs of growing modern organizations.

This isn’t new, however. Over the past four rounds of MITRE Engenuity testing, we see the same usual suspects (you know, the ones who tell you they are all #1 year after year) abusing configuration changes to boost their scores, to the tune of nearly 5-10 customer support tickets per vendor per round of recent testing. Some of the same vendors you see below are also claiming 100% automated or real-time results, which seems incongruent with their need for humans in the loop to repeatedly fix their product and redo portions of testing.

*Trend Micro chose not to participate in the first round of MITRE Engenuity ATT&CK Evaluations

VMware Carbon Black is the ONLY industry leader to require zero Configuration Changes across all four rounds of MITRE Engenuity ATT&CK testing. Other vendors may claim their security solution is easy to use out-of-the-box, but VMware is the only one to actually prove this value on one of the industry’s most respected independent tests.

Zero Delayed Detections: Why It Matters

In addition to Configuration Changes, another impediment to rapid response highlighted by the MITRE Engenuity ATT&CK Evaluation is the Delayed Detection modifier. VMware Carbon Black achieved 100% real-time detections on this year’s round of testing, meaning zero Delayed Detections. At least two other vendors we already showed above exhibited Delayed Detections for over 10% of their results, which should be kept in mind when assessing the legitimacy of their analytic coverage or visibility scores.

According to MITRE Engenuity’s website, for a detection to be considered delayed:

“The detection is not immediately available to the analyst due to additional processing unavailable due to some factor that slows or defers its presentation to the user, for example, subsequent or additional processing produces a detection for the activity. The Delayed category is not applied for normal automated data ingestion and routine processing taking minimal time for data to appear to the user, nor is it applied due to range or connectivity issues that are unrelated to the capability itself. The Delayed Detection modifier will always be applied with modifiers describing more detail about the nature of the delay.”

MITRE goes on to say they provide these modifiers to allow “end users to weigh, score, or rank the types of detection against their needs.”

A product’s ability to automatically detect sophisticated adversary TTP out-of-the-box without a human in the loop is critical to give organizations a fighting chance against the modern threat landscape. These details might not make for catchy tweets and press release headlines, but they mean a world of a difference to SOC teams trying to advance their security posture with the finite resources at their disposal.

Putting It All Together

In a world where scalability and automation are crucial to keeping sophisticated adversaries at bay, today’s organizations need to modernize their SOCs with future-ready solutions that provide maximum security value from Day 1. Don’t settle for starting behind the 8-ball with a vendor whose product can’t stand up to today’s threats without some serious expert intervention.

VMware empowers SOC teams to go toe to toe with even the most advanced adversaries out there while taking the burden off the analyst via authoritative context and actionable insights. VMware is changing the game for SOCs and increasing their operational confidence. We do this through the precise combination of world-class threat research from the human expertise we offer via the VMware Threat Analysis Unit, together with the powerful AI analytics we use to stop attackers in their tracks.

We strike this balance of humans and technology for the most holistic approach to turning telemetry into insight. Security through VMware reduces time to resolution via connected control points that are built into the very fabric of your infrastructure instead of being bolted on as an afterthought. Our future-ready security offerings extend our industry-leading EDR capabilities beyond the endpoint to secure your network, endpoints, workloads, and containers with the option for native Managed Detection and Response services for extra assurance. At VMware, we don’t just meet the customer where they are in terms of security maturity, we act as the vehicle to move them from where they are to where they want to be, relentlessly shifting the advantage back to the home team defenders.

To get hands-on with VMware Carbon Black visit our Threat Hunting lab environment here.

To learn more about the adversaries emulated and the relation to the evolving landscape take a quick read through 'It's Time for CISOs to decipher the Threat Actor Strategy'