I’ve spoken before on the cost of poor threat intelligence and its risk to an organization. I’ve also spoken about the 4 qualities of good intelligence: relevance, timeliness, accuracy, and completeness. To better evaluate threat intelligence sources – DRIVE FOR TRANSPARENCY! If you treat threat intelligence like a black box you’re going to lose.
Here are questions to use when evaluating an external source. These are just a starting point or additions to your own list based on your unique needs.
[Relevance] Why do I need threat intelligence?
Before you go out evaluating threat intelligence sources, you need to know what you’re looking for. This is best done using a threat model for your organization and asking where threat intelligence supports visibility and decision making within that model. Remember, your own threat intelligence is almost ALWAYS better than that produced by an external source. External intelligence should complement your own visibility and reduce gaps.
Kudos: Thanks to Stephen Ramage for his comment highlighting the exclusion of such a critical question.
[Relevance] What types of intelligence are available?
Strategic country-level reporting? Cyber threats mixed with political threats? Technical indicators? Campaign behaviors? Written context? These all determine how useful, actionable, and relevant the intelligence will be for your organization.
[Relevance] Give me your context!
Make sure you understand the context provided with any data. There is a difference between threat data and threat intelligence. Intelligence helps drive effective decision-making. Context makes data relevant.
[Relevance] Which threat types?
Is it limited to botnet C2 nodes? Commodity threats in general? Does it cover targeted threats? Does the threat intelligence provide insight into your threat model?
Related Questions: How many unique threats are distinguishable in the intelligence?
[Relevance] How many direct threats to my organization or those in my industry has your intelligence identified?
Has the source ever shown direct success in highlighting threats in your industry?
[Relevance] How is the intelligence made available to consumers?
If the intelligence is not provided in a usable form, it will not be successful.
[Relevance] What types of use-cases produce the best experience/feedback? In which use cases has your intelligence failed?
This is a soft-ball question but one which should provoke a good question-answer session. The answers will illuminate their decisions developing the intelligence and highlight where the intelligence may fit best (or not fit at all).
Related question: What threat model is this intelligence attempting to address?
[Completeness/Relevance] What is the source of the intelligence?
Is this intelligence derived from human sources crawling the dark-web? Global network apertures? VirusTotal diving? This question should frame their visibility into threats and inform the types of intelligence expected. This also highlights any natural biases in the collection. Look for sources of external intelligence which complement your own internal threat intelligence capabilities.
[Completeness] What phases of the kill-chain does the intelligence illuminate?
Understand how wide, against any single threat, the intelligence goes. Does it only show C2, or will it also illuminate pre-exploitation activities as well. The wider the intelligence, the greater the likelihood of it being useful.
[Completeness] What is the volume and velocity of the intelligence?
“How much” intelligence is actually produced? Numbers don’t matter that much – but if the number is ridiculously small or ridiculously large, it is an indicator of possible issues.
[Accuracy] How is the intelligence classified and curated?
Drive for transparency in their process which helps improve your evaluation on accuracy. Be wary of “silver bullet” buzz-word answers such as “machine learning” or “cloud.”
[Accuracy] How is the intelligence validated?
Do you want to track down false positives all day? No! Do you want to rely on poor analysis? No! Make sure this question gets enough attention.
Related questions: How often is it re-validated? How are false positives handled? How can customers report false positives? What is your false positive rate? How many times in the last month have you had to recall or revise an intelligence report?
[Accuracy] Does the intelligence expire?
Expiration of intelligence is key. Is there a process which continuously validates the intelligence?
[Timeliness] How quickly is the intelligence made available to customers after detection?
Related questions: What part of your process delays intelligence availability? What is the slowest time to availability from initial detection?