I’ve spoken before on the cost of poor threat intelligence and its risk to an organization. I’ve also spoken about the 4 qualities of good intelligence: relevance, timeliness, accuracy, and completeness. To better evaluate threat intelligence sources – DRIVE FOR TRANSPARENCY! If you treat threat intelligence like a black box you’re going to lose.
Here are questions to use when evaluating an external source. These are just a starting point or additions to your own list based on your unique needs.
[Relevance] Why do I need threat intelligence?
Before you go out evaluating threat intelligence sources, you need to know what you’re looking for. This is best done using a threat model for your organization and asking where threat intelligence supports visibility and decision making within that model. Remember, your own threat intelligence is almost ALWAYS better than that produced by an external source. External intelligence should complement your own visibility and reduce gaps.
Kudos: Thanks to Stephen Ramage for his comment highlighting the exclusion of such a critical question.
[Relevance] What types of intelligence are available?
Strategic country-level reporting? Cyber threats mixed with political threats? Technical indicators? Campaign behaviors? Written context? These all determine how useful, actionable, and relevant the intelligence will be for your organization.
[Relevance] Give me your context!
Make sure you understand the context provided with any data. There is a difference between threat data and threat intelligence. Intelligence helps drive effective decision-making. Context makes data relevant.
[Relevance] Which threat types?
Is it limited to botnet C2 nodes? Commodity threats in general? Does it cover targeted threats? Does the threat intelligence provide insight into your threat model?
Related Questions: How many unique threats are distinguishable in the intelligence?
[Relevance] How many direct threats to my organization or those in my industry has your intelligence identified?
Has the source ever shown direct success in highlighting threats in your industry?
[Relevance] How is the intelligence made available to consumers?
If the intelligence is not provided in a usable form, it will not be successful.
[Relevance] What types of use-cases produce the best experience/feedback? In which use cases has your intelligence failed?
This is a soft-ball question but one which should provoke a good question-answer session. The answers will illuminate their decisions developing the intelligence and highlight where the intelligence may fit best (or not fit at all).
Related question: What threat model is this intelligence attempting to address?
[Completeness/Relevance] What is the source of the intelligence?
Is this intelligence derived from human sources crawling the dark-web? Global network apertures? VirusTotal diving? This question should frame their visibility into threats and inform the types of intelligence expected. This also highlights any natural biases in the collection. Look for sources of external intelligence which complement your own internal threat intelligence capabilities.
[Completeness] What phases of the kill-chain does the intelligence illuminate?
Understand how wide, against any single threat, the intelligence goes. Does it only show C2, or will it also illuminate pre-exploitation activities as well. The wider the intelligence, the greater the likelihood of it being useful.
[Completeness] What is the volume and velocity of the intelligence?
“How much” intelligence is actually produced? Numbers don’t matter that much – but if the number is ridiculously small or ridiculously large, it is an indicator of possible issues.
[Accuracy] How is the intelligence classified and curated?
Drive for transparency in their process which helps improve your evaluation on accuracy. Be wary of “silver bullet” buzz-word answers such as “machine learning” or “cloud.”
[Accuracy] How is the intelligence validated?
Do you want to track down false positives all day? No! Do you want to rely on poor analysis? No! Make sure this question gets enough attention.
Related questions: How often is it re-validated? How are false positives handled? How can customers report false positives? What is your false positive rate? How many times in the last month have you had to recall or revise an intelligence report?
[Accuracy] Does the intelligence expire?
Expiration of intelligence is key. Is there a process which continuously validates the intelligence?
[Timeliness] How quickly is the intelligence made available to customers after detection?
Related questions: What part of your process delays intelligence availability? What is the slowest time to availability from initial detection?
Stephen Ramage
The other key question, throughout all of this – What exactly am I trying to achieve with this, and how does this intelligence help me achieve that goal?
Sergio Caltagirone
Great comment Stephen! I added it to the list of questions above – initially I assumed that any organization going out looking for threat intelligence would have their own use cases defined, but you’re right that this may be unrealistic.
Stephen Ramage
I was primarily thinking of cases of Rabbit-hole-itis, where you spot an interesting looking article, and chase it down, then start following the wrong chains.
Kind of like what happens when you look something up on Wikipedia, say, some musician or other, and before you know it you are reading up on key stages in the making of ingredients for nuclear weapons, and have no idea how you got there.
Robert Peeler
It is important to realize that evaluating intelligence should be accomplished from a requirements perspective. Intelligence should be used to fill requirements that are based on your information gaps. These gaps then become Priority Intelligence Requirements (This implies prioritizing requirements but that is another topic all together). That is how you start to evaluate intelligence products–does it answer the questions you need answered? If it doesn’t answer your requirements, then it isn’t needed. You have to know what you know and what you need to know in order to determine if the intelligence is of value to you and your organization. Another key concept to keep in mind is the requirements need to support leadership and operations at the strategic, operational and tactical levels of your organization. I’ll stop there to keep it brief.