ATT&CK™-Based Product Evaluations

October 23, 2017
Threat-based Defense: Post by Frank Duff, Chris Korban, and Cody Thomas
Frank Duff, Chris Korban, and Cody Thomas standing outside

The Endpoint Detect and Respond (EDR) market, as defined by Gartner, is now valued at $500 million, a number that doubled from 2015 to 2016. Unlike anti-virus software, which focuses on preventing exploitation, EDR products attempt to detect malicious post-exploit behavior. Cybersecurity vendors are now aligning with cybersecurity experts who have been saying: we cannot prevent adversaries from getting into the network. As evidence, just look to Mandiant, who has estimated the median time an adversary is on a network for 99 days before detection.

The EDR market has given opportunity to both new startups and existing traditional security vendors that have focused on prevention (e.g., next-gen antivirus, application isolation). There are an overwhelming number of choices for enabling post-exploit detection and behavioral monitoring. What’s the best way to navigate these technology choices, especially since there is no unified way to express what exactly these EDR tools can do?

The ATT&CK™ knowledge base provides a common foundation for describing both testing criteria and results. ATT&CK is a MITRE-developed, globally-accessible knowledge base of adversary tactics and techniques based on real-world observations of adversaries’ operations against computer networks. We’re using ATT&CK and its associated Cyber Analytics Repository to help our government sponsors make informed decisions around what threats a specific EDR tool is best at detecting.

Using ATT&CK to Evaluate EDR Capabilities

MITRE runs a skunk works that we call LETS (because what good is a skunk works project without an acronym?) that stands for “Leveraging External Transformational Solutions.” We wanted to tackle the EDR capability evaluation problem by focusing on a series of lightweight, yet targeted, evaluations of EDR tools being used or considered across the government. Drawing upon internal MITRE research and expertise, we wanted to use LETS to not only help the government, but also to raise the industry’s capabilities and their ability to articulate their functionality to customers by using the common language from the ATT&CK knowledge base.

To begin to articulate an EDR tool’s post-exploit detection and behavioral monitoring capabilities and gaps, we picked a specific threat actor of interest to our sponsors, and then executed the ATT&CK techniques associated with it in a cyber exercise.

We chose APT3, whose post-exploit behavior relies on harvesting credentials, issuing on-keyboard commands (versus Windows API calls), and using programs already trusted by the operating system (living off the land). APT3 does not tend to do elaborate scripting techniques, leverage exploits after initial access, or use anti-EDR capabilities, such as rootkits or bootkits. This made APT3 emulation a perfect starting point for evaluating an EDR tool’s capabilities.

Adversary Emulation Planning for EDR Evaluation

APT3 can be emulated using three general phases: initial compromise/setup, network propagation, and collection/exfiltration. To give a general sense of the emulated behavior, these phases are briefly described in terms of the adversaries’ intended goal, but more importantly using how they achieve these goals to articulate common test criteria for evaluating EDR tools:

Phase 1 – Initial Compromise

The goal of the Initial Compromise phase is to achieve successful code execution and control of a system within the target environment. APT3 primarily conducts initial compromise using spear-phishing, delivering implants through both malicious attachments and malicious links, or through browser-based 0-days on a compromised website. This phase also includes getting the command and control infrastructure set up and making sure that the implants are properly encoded and protected.

Phase 2 – Network Propagation

The goal of the Network Propagation phase is to identify and move to desired systems within the target environment with the intention of discovering credentials and documents for exfiltration. The following are example ATT&CK techniques APT3 uses to achieve the related tactics for Network Propagation.

  • Privilege Escalation – Legitimate Credentials (T1078), Exploit Vulnerability (T1068)
  • Peristence – Accessibility Features (T1015), Start Folder (T1060), New Service (T1050), schtasks (T1053), Legitimate Credentials (T1078)
  • Credential Access – Credential Dumping (T1003), Credentials in Files (T1081), Input Capture (T1056)
  • Discovery – Permission Groups Discovery (T1069), Account Discovery (T1087), System Network Configuration Discovery (T1016), System Network Connections Discovery (T1049)
  • Lateral Movement – Windows Admin Shares (T1077)
  • Remote Copy and Execution – schtasks (T1053), Remote Services (T1035)

Phase 3 – Exfiltration

The goal of the Exfiltration phase is to identify documents of potential value, stage them, encrypt them, and get them out of the target environment. This is potentially the noisiest section and is often saved for last after making sure that there is sufficient access, persistence, and redundancy within the target network.

Tackling Evaluation of Specific EDR Tools

After establishing a common set of test criteria, we engaged with vendors offering solutions that our government sponsors were considering, evaluating, or deploying. Several vendors wanted to engage with this MITRE project, primarily because it would not only help them articulate their capabilities, but also provide them with a standard measuring stick from which they a way to differentiate themselves in a crowded marketplace. 

Vendors were provided access to a virtual cyber range in which, with help from the vendor, MITRE deployed their EDR tool. MITRE then provided the adversary emulation red team, while the blue/hunt team was split between MITRE and the vendor. We limited the emulation to one or two days per vendor engagement. While the condensed activity results in less stealth, and is atypical of APT intrusions, it still enables us to articulate the detection and prevention capabilities of the evaluated technologies.

MITRE created a report for each EDR tool, outlining the emulation event from deployment to execution. This report is shared with the vendor, who may then use it to inform their technical roadmaps. At the same time, we provide the report to the MITRE sponsors who have the vendor on their roadmaps. This helps our sponsor better understand a vendor’s capability to address a known threat. Included in the report is a mapping of the evaluation and tool’s ability to address the Advanced Persistent Threat (APT) to the ATT&CK Matrix. See Figure 1.

Figure 1: Example APT3 emulation ATT&CK results.Grey - APT3 techniques not tested, Green - tested and detected, Yellow - tested and weren't detected but could have, and Red - sensor gaps

Expanding ATT&CK to Cover More EDR Tools and Threats

At this time, the MITRE’s LETS program is set to add to the APT3 emulation evaluation with an APT29 adversary emulation, which allows for greater coverage in sophistication of TTPs leveraged. APT29 TTPs heavily feature more sophisticated techniques and high levels of scripting. We believe that APT3 and APT29 will address the scope of detection capabilities across APT variances and an EDR’s capability to detect APT post-exploit behavior.

The ATT&CK knowledge base is used to map against real-world threat behavior, describing both testing criteria and results. By participating with MITRE in the LETS program, EDR vendors receive impartial feedback that can positively impact their capability roadmaps. Because MITRE serves as a not for profit government agent, its sponsors also get an unbiased assessment of the tool’s performance specifically related to that APT and mapped to ATT&CK techniques. Through multiple engagements, the entire EDR industry has a chance to learn and implement the same practices. The community at-large too will benefit as MITRE expects to make these adversary emulation plans available on the ATT&CK website in the coming months.