Evaluating Cyber Threat Intelligence Services, Part TwoOctober 3, 2016
My last post focused on evaluating cyber threat services themselves. In this post, I'll delve into additional considerations as you choose which vendor to partner with.
What exactly are you buying?
Vendors offer different products and services, so make sure you understand everything that’s included in the service's subscription before you purchase it. For example, some subscriptions include access to analysts who perform a range of services--like analyzing malware on samples that you provide or answering questions about reports—that you may not need.
One way to know what you're buying is by testing the service. Some vendors offer a free trial period so you can evaluate whether the product easily fits into your existing network defense flows or other applications, so be sure to ask the vendor about this.
How will the vendor track and use your organization's queries and submissions?
An important consideration when evaluating a threat intelligence service is how the vendor tracks and uses any queries or information your organization enters in the service. Some vendors may aggregate information and use it for trending or analysis before disseminating it to other customers.
Although vendors make efforts to protect customers' privacy and anonymity, consider your own operational security needs. For example, if your organization identified a malicious callback domain from malware you detected on your network, and an analyst then searches for that domain on the threat intelligence service, the vendor may be able to surmise that your organization has been affected by that malware.
These operational security concerns may affect how your organization uses the service. This may affect whether analysts should submit new raw intelligence data such as malware samples directly to the service or limit activity to searches of indicators.
How do vendors collect, evaluate, and analyze threat information?
Vendors often consider their sources and methods of gathering intelligence to be proprietary, but they should be willing to describe in general terms how they collect, evaluate, and analyze threat information used to produce their finalized intelligence.
Find out what types of sources the vendor uses to obtain threat information. Examples might include publicly available malware databases, passive DNS and WHOIS data, and security appliances. Ask how the vendor processes and analyzes the threat information to ensure that it's reliable and accurate.
If the vendor provides attribution to specific threat actor groups, find out how that’s accomplished and whether confidence levels will also be provided. Ask what general methodology they use in conducting their analysis and forming their conclusions. In some cases, vendors may be staffed by former military, law enforcement, or intelligence community professionals, so they adopt formalized methodologies and terminology developed within those communities, like the Analysis of Competing Hypotheses, Words of Estimative Probability, and the Diamond Model for Intrusion Analysis. In other cases, vendors rely on in-house or less formalized methodologies.
Knowing how the vendor performs its analysis is important so that you can better assess the quality and confidence level of the reporting and the meaning of the alerts you receive from your sensors when you make use of the indicators.
You should also ask how the vendor conducts quality control of the atomic indicators it provides, such as ensuring that IP addresses of hosting services or benign scanners aren’t included in the service (or are at least marked as such). If a vendor isn't properly vetting indicators, and your organization ingests them for alerting on your security appliances, it could cause excessive false positives for your analysts to research.
Wrapping it up
Cyber threat intelligence services can provide a rich source of actionable cyber threat indicators to aid in intrusion detection and response. So do your due diligence and pick the right service (and vendor) for you.