15 Things Wrong with Today’s Threat Intelligence Reporting

What I think when I read most threat intelligence reporting

As I have written before, intrusion analysis is equal parts knowing the technical elements of an intrusion and being an analyst.  However, most in this domain spend an inordinate amount of time studying technical details compared to honing any analytic skills.

How long has it been since you’ve taken a highly technical course?  (probably within the last year or two)  How about an analysis course?  (probably in the last 5 years, 10 years, never?)

I read several threat intelligence reports daily.  It is painfully obvious how the lack of analytic skill is harming the discipline. Many folks come from technical degree backgrounds and analyze packets and binaries well enough but can’t seem to tell the difference between inductive, deductive, or abductive reasoning.  Furthermore, their managers and mentors never recognize a problem, they just send them to more technical courses.

What is the risk?  Threat intelligence provides insight and context to improve decision making.  The risk of bad intelligence is high. Bad decisions can easily be made from poor intelligence – potentially doing more harm than good.  Good analytic practices improve analysis thereby decreasing the risk of poor intelligence.  You could have the best packet analysis skills in the world, but if you cannot communicate your conclusions effectively to those who need to act on your information those skills are effectively useless in threat intelligence.

We need to do better.  I started this post about a month ago and wrote down a “lesson” whenever I saw an example of poor analysis.  Needless to say, I saw some of these several times.  (Contrary to the recommendation of others, I will not cite/quote specific examples – I believe that would only name and shame others)

Trend – the word actually means something

How many times per week must I read about a new “trend” from threat intelligence?  One or two events does not constitute a trend.  Even three or more events, depending on the universe of events, may not constitute a trend.  Trends are serious.  True trends in adversary activity and methodologies inferred by threat intelligence should drive data collection, analytic tradecraft, and defensive decisions.  Before you start throwing out the word trend just because you’ve seen something a few times, consider the millions of other events you’re not seeing and consider if they’re just an anomaly rather than a trend.

Analysts and conclusions are like horses: sometimes you need to lead them to water

In many cases I can follow the logical progression of hypotheses and facts to the conclusion.  In some cases I cannot.  Either because an analyst failed to include the appropriate evidence/fact on which now an assumption must rest or because of convoluted logical reasoning.  Ensure evidence supports your conclusions and the logical reasoning is clear.  Don’t assume that what is clear in your mind will be clear in mine.

You can’t be completely confident all of the time – use words of estimative probability

Do you know how often I see the effective use of estimative probability in recent threat intelligence reporting?  Almost never.  This is a problem.  Not everything presented is irrefutable fact; in fact, a good analysis will have a proper mix of data/fact, hypotheses and conclusions.  The confidence values of these conclusions vary.  When you don’t effectively apply estimative probability and variable measures of confidence it removes value from the analysis and increases the risk of poor decision making by consumers.  First, if you don’t know what estimative probability is, LEARN about it.  Then learn how and when to apply it properly. Importantly, also know what words/phrases to avoid (i.e. weasel words).

Never be afraid to include contrary evidence

Do you know how many times I saw evidence contrary to the conclusion presented in a threat intelligence report this month?  Never.  Practice analytic honesty.  If there is exculpatory evidence, contrary evidence, or an alternative hypothesis – show it.  As long as you’re following some of the other lessons here (e.g., separating fact and hypothesis, using words of estimative probability) it will strengthen your analysis and provide more value to the consumer.

Just because you’ve seen something for the first time doesn’t mean it’s the first time it happened

We all love finding something awesome and telling the world.  It’s cool because we all want to know what you’ve found!  But, please don’t assume it is the first time it has happened or even the first time it has been seen.  Having confidence is critical, but hubris is deadly to analysis.

Don’t operate on an island

You are not alone!  Don’t act like it.  Share and consume, enrich and enhance.  Go ahead and build on the analysis of others (citing appropriately).  Whatever your observation point or data sources, they’re not omnipresent.  I rarely see analysis reference other (obviously) related pieces.  How is that?  The power of defenders lies in our community and our ability to work together against an adversary.

Be bold, but don’t be stupid

I like my analysis like I like my coffee: bold.  But, there is a line between taking facts to their logical conclusion and taking facts to crazy-land.  The difference is logic.  Ensure your conclusions and hypotheses follow logically from the facts through induction, deduction, or abduction.  If your conclusions cannot be logically traced or tested, then they’re likely living in crazy-land.

Don’t mix hypotheses, conclusions, and facts

Hypotheses, conclusions, and facts are not the same.  Your intelligence reports should not treat them as such.  Ensure that your readers can effectively separate the three through your use of language, formatting, etc.  When the three are confused it can lead to erroneous assumptions by consumers and lead to decisions made on weak conclusions rather than facts.

Save hyperbole for the glossy promotion material

Hyperbole has its place.  It doesn’t have a place in threat intelligence.  Save that for the glossies.  Be precise, honest, and accurate.  Don’t embellish or exaggerate.  Trust me when I say we have enough people running around like chickens with their heads cut off in this field.

Logical fallacies are just that, get to know them

Enough said.  I’m sorry I have to say this, but please understand the differences and applicability of deductive, inductive, and abductive reasoning BEFORE writing your first threat intelligence report.  Or, at the very least, have an editor who knows the difference.

Don’t create new words when existing words suffice

I’m not going to name-call here.  You know who you are.  There are times when words/phrases have multiple meanings.  I understand that.  But, aside from that….stop it.

Tell a story!

Your analysis is a story.  You’re effectively documenting history – studying the past – in the hopes of making conclusions and judgments which will help the present and future.  While you are documenting the activity of computers you are ultimately describing the actions caused by adversaries.  Just like any story your report should have a beginning, middle, and an end.

Answer my questions

Write as if you are in a conversation.  Think about what somebody else may ask of what you’re saying, and address those questions in the text.  Any questions left unanswered have the ability to form into assumptions on the part of the consumer/customer.  If you don’t have an answer, feel free to write: no further information.

Be concise, be accurate

Practice analytic honesty and respect the time of your reader.  The report you’re considering may actually need to be three different reports – one describing all of the malware reverse engineering, one describing all of the network activity, and another describing the threat itself which references the other report.  Putting everything in one report does not make it more consumable, it makes it less consumable and allows analysts to muddle up various lines of analysis.

Describe diagrams, charts, and tables both in the narrative text but also in a caption

This is just a pet-peeve of mine, but one which I find increases the readability of threat intelligence reports.  Make sure you describe your diagrams, charts, and tables in both the narrative text (as part of the story) and also in a caption.  I find this necessary because as I move backwards and forwards through a report reading and re-reading, forming and re-forming logical chains, I don’t want to hunt for the description in the text every time.  I also don’t want to jump to the caption in the middle of text if not necessary which breaks my concentration.