Ramblings: Threat Hunting And Forensics Are Different.

By Matthew Demaske, Director of Threat Research

Just a little Friday rant before I kick off the weekend.

There is a huge difference between a traditional computer forensic investigation and threat hunting.

Let’s put it in a law enforcement prospective. Forensic analysis happens after someone reports items being stolen. Diamonds at a store. Your living room TV. Your collection of vintage M*A*S*H figurines. Coming in, you already have one piece of important information. Confirmation that something is wrong. Stuff is either missing or it isn’t. Binary answer. It’s 1 or a 0. A detective’s task is to find out the how, why, when, and most importantly, the who. They’ll dust for prints, interview neighbors, and search local pawn shops.

However, with threat hunting, the law enforcement analogy is like trying to find a robbery in progress. Your criteria is no longer binary. You don’t have any confirmation. Something may be happening or it may not be happening. You have to rely on the quality of information you’re gathering from various sources and your own experience. Are you tracking alarms? Are you focusing on places with poor security? Places that have been breached in the past? A specific time or day of the week robberies will likely happen? Where are you dedicating most of your resources? The vault at the top of the super secure government building or the local bank with an 82 year old security guard? Maybe you see an alarm being triggered at a jewelry store. Does that definitely mean it’s being robbed? No. You have to investigate further.

When you’re threat hunting, you often have a ton of data coming in from various sources and you have to make an educated guess as to what looks suspicious enough to investigate further. With a typical forensics investigation, you at least know something HAS occurred. Forensics/IR teams investigate incidents. Hunters are looking at events. The difference? According to ISO/IEC 27001, an information security event is “an identified occurrence of a system, service or network state indicating a possible breach of information security policy or failure of safeguards, or a previously unknown situation that may be security relevant.” An information security incident is “a single or a series of unwanted or unexpected information security events that have a significant probability of compromising business operations and threatening information security.”

Depending on how that incident was discovered, there could be a ton of context for the investigation team right off the bat. Forensic investigators often have the luxury of having access to the affected system(s), with the acquisition of disk or memory images, too. That doesn’t happen as a threat hunter. You get bits and pieces. For instance, if you obtain a memory image, you can run it through Volatility and get a ton of great information back. PSTree is awesome. Can you get that kind of in-depth memory analysis streamed in real time to analysts 24/7, 365? Not that I’m aware of. Even if you did, the sheer amount of data would be staggering as the contents of memory are constantly changing. There are bandwidth and storage costs to be considered if you’re sending this stuff off system to a SIEM or log aggregation suite. It’s the primary catch-22 for a threat hunter. You want as much data as possible, but that comes with the increased time it takes analyzing the data.

My point is that you have to be picky with your definition of “that’s bad” when it’s an after-the-fact investigation, especially if there will be legal ramifications. No one wants a forensic expert on the witness stand saying words like “maybe”, “possibly”, or “more probable than not”. They want binary answers. Yes or no. The best thing we can do with threat hunting is “There’s a good chance that’s bad, but we need to investigate it further.” Hunters find suspicious and the IR/Forensics team confirms it. If there’s no IR team, the hunter takes off their hunter hat and puts on their forensics hat. Either way, there’s a tactics shift.

I had an exchange with someone in the forensics field over the context in which I discussed a possible indicator of compromise. They were looking at it with their forensics hat on and accused me of being inaccurate with description of the artifact. However, I wasn’t describing the indicator of compromise as if I were a forensics investigator. I was describing it as if I were a threat hunter looking for suspicious activity in a sea of benign log data.

Take Powershell for example. Powershell executions aren’t inherently suspicious on their own. You’ll probably see it a million times if you’re ingesting basic process execution logs from Windows hosts(event ID 4688). But, if you’ve got access to that host’s entire event log history while investigating an incident, you can look at the context in which powershell.exe was ran. You would then take the “Creator Process ID” field data and start building out parent/child relationships of processes. That would absolutely confirm the presence of an embedded macro running. Word.exe spawns powershell.exe which then spawns bla bla bla and so forth. If these logs are coming into an an aggregation engine, you may not be able to create those parent/child relationships on the fly.

So, if I’m telling a fellow hunter how to look for malicious macro activity, I would say “look for word.exe and then powershell.exe executing soon after. That may be worth investigating if it’s on the same host”. However, a forensic investigator would say “That’s not accurate. The indicator would be word.exe being a parent of powershell.exe, not those executions happening around the same time frame. It could be a coincidence.” The forensics person is absolutely correct, but it’s irrelevant to what my analysts are seeing coming over the wire.

Of course, if you’re using a program such as Sysmon, you will able to see stuff like parent/child relationships and command line data, so my argument would be moot at that point. Your analysts will be able to see that stuff, too.

Neither of us were wrong. We were just looking at it from different perspectives. Threat hunting is still somewhat new under the cyber security umbrella and these mix-ups will happen. As time goes on, let’s hope they become less frequent as the two groups have more interaction.

Have a good weekend!