Detection is the Horse, Investigation is the Cart – Use in That Order

I received some interesting responses from my last week’s post (Incident Detection, Then Incident Response) so let me try to answer them all collectively.

No, my post was not a knock against incident response (IR) or forensics tools.  I believe we are getting things out of order.  It is about detection first.  Better analysis? Good. Better Response. Good. But it all starts with breach detection.  In fact, if we had better breach detection, organizations would actually get more value out of their IR/forensics tools.

The inability of organizations to detect breaches is easily explained.  The picture below is my attempt to illustrates what I call The Breach Detection Gap.  This gap exists  between the numerous layers of prevention solutions and IR/forensics tools leaving organizations unable to detect breaches at the point of infiltration.

The IT security  market has been fixated – technically and emotionally – on prevention. Hence the numerous “usual suspects” on the left side of the breach.  I think my position is clear (cystal) that a prevention-centric strategy is doomed to failure.  Tradecraft relentlessly and rapidly evolves to evade any gains in prevention, and targeted attacks and the Advanced Persistent Threat are engineered to evade the specific defenses meant to defend their target.

IR and Forensic tools provide deep insight and valuable analysis to the breach investigation process, but are only brought to bear after the breach is detected.  Unfortunately, this is where most organizations spend the meager budget slice that is set aside for post infiltration.

The Breach Detection Gap is the critical exposure between prevention tools and IR/forensics tools that leave organizations without the means necessary to detect breaches in real-time.  Obviously, without detection there can be no timely response.  Which is my point of last week’s post: re-packaging IR tools as the solution for breach detection problems is not the answer.  The answer must start with faster and more accurate detection.

Someone also asked why I don’t name names.  I try to write this blog to stimulate thought and while I unashamedly say where Triumfant solves specific issues I try very hard to keep this from being an ongoing advertisement.  I also have never believed that there is any value from directly speaking in a negative manner about any other vendor.  There are some good IR/forensics tools in the market that are very hot right now, and when products get hot, the market begins to act strangely around them.  My post was not a knock on those products, but on the efforts I see in the market to position those tools with professional services as the solution to the Breach Detection Gap.  Make no mistake, the organizations around these hot products and event the vendors behind these products see this as a chance to sell professional services projects to hunt down breaches.  I will leave it to you to figure out who those vendors are, but I think in most cases the answer will be easily discerned if organizations resist the hype.

What I did not say in last week’s post is that Triumfant is positioned to detect breaches in real time.  There are ample posts that address that directly as well as a new whitepaper on our site, so I won’t go into details here.   I will say that while heuristics, behavioral, and IPS/HIPS are also being directed to the problem, I think that Triumfant’s use of change detection and the analysis of change in the context of the host machine population is uniquely suited for the role of breach detection.  You get rapid detection (real-time), and within minutes we provide detailed information to help formulate an informed response, and we custom-build a remediation to stop the attack and repair the machine.  That is rapid detection and response.

And while Triumfant provides a wealth of IR/forensics data, we fully endorse the use of IR/forensics tools to provide the full range of post-breach investigative work.

But it all starts with detection.

Incident Detection, Then Incident Response

There seems to be an interesting and, I believe unfortunate, trend emerging in IT security:  Incident Response (IR) and Forensics tools are being wrapped in professional services and being sold as the solution to the breach detection problem. While I am happy that there is growing understanding that there is a breach detection problem, the reaction to that recognition is disappointing and misses the mark.

I think the point is obvious and is right there in the name “Incident Response”.  Response is not detection.  It is a step after detection – 1. Detect the problem. 2. Analyze the problem. 3. Fix the problem.  You could group #2 and #3 as respond, but they still follow detect.

You see, I thought detection was the issue.  While coming up with faster and more efficient ways to respond is laudable, I did not think what we needed was a better response to breaches that go undetected for an average of 173.5 days (Trustwave Report).  Just to make sure I was not missing something, I reviewed all of the excellent breach investigations and reports (Verizon Business, Trustwave, IBM X-Force, and Mandiant).  While some note that the time from detection to containment, but it is certainly not the focus.  The consistent focus I take from my reading is that organizations are getting breached and are not prepared to detect those breaches.

Unfortunately, there are several organizations making hay with selling professional services engagements under the umbrella of incident response.  The IT security market has a long history of seeing success and extrapolating that success into a rush to copy that success.  This is one of those cases.  Then marketing kicks in and the opportunity for the market to take constructive steps forward is squelched by the vendors rushing toward the next pot of gold, and organizations being swept into the hype.  Then these same reports will come out next year and there will be collective head scratching as to why the numbers have not improved.

The winner is the adversary, who is quite fine with 173.5 days of undetected access to organizational networks.

A simple analogy is firefighting.  Firefighters diligently and continuously train to better respond to a fire when called.  There are constant technological breakthroughs in equipment that also help them respond to a fire when called.  All of that training and equipment is put into use when they are called (the fire is detected).  Firefighters are not responsible for detection, they are all about the response. And while I am not a firefighter, my guess is that firefighters would tell you that the sooner the fire is detected, the better their response.  I would also guess that rapid detection is a key component to reducing loss.  Having a better, more expensive fire investigator will not reduce loss.

The first step to solving the breach detection problem is deploying tools that rapidly detect breaches at the point of infiltration.  Studies prove that prevention tools cannot provide that detection, and IR/Forensic tools are not built for detection.  Detection must be addressed first.  Then you can deploy all of these marvelous response offerings.

Another explanation is that organizations have twisted themselves into a really unfortunate Gordian knot. Maybe they are just beginning to understand the problem, but have reconciled that they will take action if and when then are breached.  This is not a good strategy, because statistics say it is likely they already have been breached, but simply don’t know it yet because they lack the tools to detect breaches.   There is no more “if”, and the “when” has likely already happened.  That is not FUD, that is what the statistics say.  Once a breach is detected  – the statistics say that 92% of those breaches will be detected by a third party and not the breached organization – then they will spend enormous amounts of money to have someone come in and do lots of expensive analysis and make recommendations that they will likely ignore.  The organization of course must deal with the financial, regulatory, and reputational effects of the 173.5 days the adversary had access to their confidential data and intellectual property.

To paraphrase a quote from Churchill I have used before, people frequently stumble over the truth; unfortunately, they often pick themselves up and carry on as if nothing happened.  I fear this is one of those collective moments when organizations have stumbled onto the truth and will not be the better for it.

In 10 Days, the Mac Safe Haven Becomes a Botnet Spewing, APT Vulnerable OS

In rapid succession, the IT security world, not to mention the perceived cocoon of safety for Mac users, was rocked by two announcements.  On April 4, Russian antivirus company Dr. Web announced that they had discovered a Mac Botnet, called Flashback, and that the bot had infected 600,000 machines.  About ten days later, Kaspersky announced the discovery of a backdoor trojan called Backdoor.OSX.SabPub.  This attack leverages an exploit that uses malformed Word documents to deliver malware that opens a backdoor that can be used for advanced, persistent attacks.  Holy APT Batman!  Perceived safety to botnet to advanced persistent threat in 10 days!

Oh the shame.  The Mac went from safe haven to botnet spewing, APT exploitable platform tied to three-year old vulnerabilities before our very eyes.  As I tweeted, the heads of the Mac fanboys and the APT crew were simultaneously exploding.  Mac users were sent to various sites to download software to check their machines for Flashback like common Windows XP users.  I could not help but wonder if some enterprising bad guys had set up malware delivery disguised as Flashback checkers – wouldn’t that have been ironic.

I am really just having some fun here.  I take no joy in the Mac becoming a target, although it is good for business.  I am also not on some war against “smug” Mac owners because I have made the jump myself.

For me, the folklore/mythology of the Mac world as a safe haven from malicious attack reminds me of a scene from the classic movie and personal favorite, Butch Cassidy and the Sundance Kid.  In this scene, Butch and Sundance have fled to Bolivia and have taken a legitimate job guarding the payroll for a mining company.  At the beginning of the scene they are riding with the old, hardened mine boss (played perfectly by the great character actor Strother Martin) and begin to argue where the inevitable ambush will occur.  The mine boss responds disdainfully: “Morons. I’ve got morons on my team. Nobody is going to rob us going down the mountain. We have got no money going down the mountain. When we have got the money, on the way back, then you can sweat.”

Mac users, I hope you have enjoyed the ride down the mountain.  The recent Mac malware news just means that the downward portion is over, and now that there is a critical mass of Macs plugged into the networks and systems where the money lies. It is time for Mac users to sweat.

We could engage in what I am sure will be an animated conversation about the superiority of the Mac OS and the inherent vulnerabilities of Windows, but I contend this was all about opportunity.  Sure Windows machines were likely the road of least resistance, but malware writers have proven to be a resilient and industrious bunch and repeatedly rise to find a way around every barrier put in their path.  So now that the opportunity has arrived – what the adversary wants is on or accessible via the Mac – the Mac OS barriers will also be breached.

I should point out that Mac users are not finished with their journey into the seedy underbelly of IT security.  Not surprisingly, the sales of Mac AV software has gone way up.  Wait until the Mac people connect the dots that the same crew that discovered the malware also sells them AV software.  Of course, that AV software will at least partially return their cocoon of safety, until they find out that motivated adversaries will drive around their new shiny AV software like a traffic cone on the interstate.

I hope they enjoyed the ride down the mountain.

Digitally Signed Malware Proves Again That Attacks Get Through Your Shields

So what, Triumfant guy, exactly gets through my shields?  You tell me I will be breached and you give me statistics, but I have AV, whitelisting, deep packet inspection, and every other acronym and buzzword in place. Oh yea, and I have “the cloud” (pause for tympani emphasis) providing me prevalence information and other cloud-based stuff.

Well, digitally signed malware gets past your protections.  Not according to me, but according to several sources – Symantec, Kaspersky, AlienVault and BitDefender – cited in a March 15, 2012 PC World article “Digitally Signed Malware Is Increasingly Prevalent, Researchers Say”.

It is the blackhat version of “these are not the droids The Droids You Are Looking Foryou are looking for”, using the certificates to get the malicious code waved through.  Some of the first evidence of this technique was found in 2010 in the analysis of Stuxnet.  The PC World article provides evidence that the technique is showing up with increasing frequency.  The article tells in good detail how it works and what protections it can evade, including whitelisting.

This technique is illustrative of the ongoing battle between good and evil in IT security.  Operating system advances in Windows 7 and other OS versions were thought to advance the security of systems, and the adversary then takes the very techniques used to make the systems more secure and subverts them to find new ways to deliver malicious code and evade protections.  I have no interest in impugning the efficacy of prevention software and I have never said to turn off protection software.  What I have said consistently is that attacks will get through your shields.  Here is yet another example of how, and demonstrates that the adversary will always find a way to get through.  No FUD here – I would point out that every vendor cited in this story is a protection software vendor.

This story also illustrates that there are no silver bullets in protection.  Prospects often cite the use of whitelisting tools as their raison d’etre  of why they do not need something like Triumfant, but here is a clear example of how such tools are being evaded.  If you need more, there is a video from Shmoocon that shows multiple techniques for evading several whitelisting tools.  Yet another silver bullet falls short. I am not singling out whitelisting – it is just the current “It” tool of IT security.

Lastly, it is illustrative of how the foundations of trust have become less…well…trustworthy.  I have seen the validation process of a certificate authority up close, and let’s just say I am not shocked to know that malware writers can obtain certificates with false identities. With the RSA breach and other certificate authorities being hacked, the foundation of trust was already showing cracks.  Now we see examples of how trust can be subverted using this technique.

So if this technique essentially waves malware through your shields, how are you going to detect the infiltration?  That is where Triumfant fills the gap, detecting the zero day attacks and targeted attacks, including the advanced persistent threat, that infiltrate your endpoint machines and servers.

I once had a product manager from another company disdainfully tell me “when you find something that gets past my shields, you call me”.  I am looking for his number as soon as I finish this post.

Targeted Attacks Make Remote Adversaries Malicious Insiders

“Wow, your tool would be great against malicious insiders!”

This is a common conclusion made by those introduced to the Triumfant solution.  That is because instead of looking for applications or malicious executables, we detect malicious activity through change, whether a threat actor working programmatically creates the change or a malicious insider directly makes the change.

The term “malicious insider” has been gnawing at me since I delivered a short presentation for the Intelligence and National Security Alliance Innovators Showcase last week.  My new slides had several screen shots from the Poison Ivy Remote Administration Tool (RAT) that we use in demos of the Triumfant product.  It was interesting to see the reaction to those screen shots as people grasped in a very graphical way what it meant to “own” a machine.  I realized that perhaps while people have intellectually grasped what a RAT can do, they might not have fully appreciated the term “own” until they actually saw one in action. (More on RAT tools in the previous post)

Today’s attacks are not smash and grab operations – they methodically evade network and endpoint protections to establish a long-term and comprehensive presence on the machine.  These are carefully crafted incursions onto target networks that rely on persistence and stealth.

In short, they turn the outsider into an insider.  This of course is not news to those in infosec, but to the people we serve, this is an idea they are still wrapping their head around these sophisticated targeted attacks.

Once a RAT is in place, the hacker has the same access as if they were looking over the shoulder of the machine’s user.  The user literally guides them through the applications and systems on the network, providing them user IDs and passwords along the way.  This allows the hacker to spread their influence to other places in the network until they are able to access their targets.   Time is on their side, as every statistic says that they will have at least a month and on average six months to identify and exfiltrate the intellectual property or sensitive data they seek.

Attacks rarely start at the machine that holds the targeted information.  Hackers now patiently gain access to the network where they can, and then stealthily move about until they find what they need.  And new Advanced Persistent Threats like Duqu illustrate that hackers are now using sophisticated attacks to gather all manner of information to then plan their payoff attack.  As I said in the previous post, these attacks put the adversary in your boardroom, laboratories, production lines, and CFO’s office.

If six months and virtually unlimited access does not qualify the hacker as an insider, I do not know what does. Recruiting physical insiders is a long and costly process and smacks of too much Mission Impossible.  And even well placed insiders may have trouble moving outside of their areas of responsibility.  Why go through all of that risk and effort when an outsider can easily become an insider.  If the operation is discovered, the outsider simply moves to the next target.

There is another aspect to being an insider: once you are inside, all of the security measures designed to keep you an outsider are now irrelevant.  All of the carefully crafted shields an organization has in place are all pointing outward and are not equipped or designed to catch the work of an insider.  Once these shields are evaded they are no threat to the insider.  Statistics from the 2011 Verizon Business Data Breach Investigations Report say that less than 6% of data breaches are discovered by the organization’s IT shop.  That sound’s like a pretty wide gap that requires some new thinking to me.

The answer to the original question is yes, Triumfant rocks against malicious insiders.  All types.

I Smell a RAT – Breaking Into Your House to Prove a Point About Breaches

I am going to break into your house.  This is obviously a hypothetical, so there is no need to report me to the local authorities. But stay with me.

As I said, I am going to break into your house.  I can get in one of two ways.  I could use simple psychology to entice you to essentially opening the door and letting me in (social engineering) or I could use some basic information gathered about you to let me know where you are vulnerable and force my way in (hacking).  I say force, but I am a pro and in spite of your protections, if I want in I will get in and the amount of force used will be minimal.

Either way, I will break into your house undetected.

The funny thing is that once I am in, all of the money you have spent on technology to keep me out will be useless.  Not one of those technologies will be able to detect that I have evaded those technologies and am now inside.  Since I am now inside, I could turn them all off, but why bother? They are no longer of consequence to me.  The thought of that makes me chuckle as I take steps to further obfuscate my presence from the inside.

If this scenario unsettles you, I am afraid it gets worse.  Because once I am inside and have had sufficient time to cover my tracks, I am, for all intents and purposes, undetectable.  That gives me full access to your home and I will now live with you for as long as I choose.  What you see, I will see, and eventually I will know where everything in you home is, including your secret stuff.  Access to all of your accounts? Well, I was looking over your shoulder every time you logged into an account, so I have all of your IDs and passwords. When you are not home I will even have time to rummage around the house at will.  Remember that valuable thing you thought you lost? I found it.

After a while, I do not even have to watch, because you decided that all of that stuff about not using the same User ID and password for your accounts was just a bunch of scare tactics.  Anyway, even if you got the slightest bit suspicious and changed anything, I am right there and will actually watch you change your password in real-time.

If I am found, odds say it will not be by you.  You would never find me on your own.  A business partner might notice something odd, or law enforcement may get a lead on my whereabouts, but you only have a one in sixteen chance of finding me.  Even if I am found out, my average stay is about six months.  Not much more to see here anyway.

And good luck getting rid of me.  Did you think I spent all of my time eating bon-bons on the couch watching Dr. Phil? Nope. I created a little thing I like to call persistence.  There are little bits of me inside the house so if you do sweep me out I can sweep right back in.  Like those little ants that come back under your sink.  I have also used your house to control other houses I have also occupied.  After all, yours was not the first.

I write this because when we do demos, we use Poison Ivy, a generally available Remote Administration Tool (RAT) to build a RAT Trojan and take over a machine.  I am surprised to learn that this is often the first time many people see exactly what it means when a hacker owns a system.  That the hacker can see the screen, capture everything that was typed, access every application and file.  People hear about RAT tools, but in my experience, they only have an academic understanding of what it means.  Showing them firsthand gives them a very jarring emotional understanding.  If you would like to see more, we have a short (5 minute) demo video that shows exactly that.

When (not if, kids) I access your system, bypass your defenses, and install a RAT on that machine, I am by definition now a malicious insider, a topic I will expand further on my next post. I am not after Grandma’s jewels, I am after the Crown Jewels.  I am after your intellectual property and your most sensitive data.  I am looking to steal things that can set your company back financially and strategically. I am not on your couch – I am in your boardroom and in your labs and on your production line and I am watching every keystroke your CFO makes.

And I am a malicious insider with staying power.  A recent statistic published in the Trustwave 2012 Global Security Report said that on average a breach lasts 173.5 days before being discovered.  Furthermore, studies show that organizations are not equipped to discover such breaches on their own.  The 2011 Verizon Business Breach Investigation Report states that breaches are discovered by the breached organization only 6% of the time.

I would tell you to wake up and smell the coffee but you are out of coffee and you should pick up a gallon of milk while you are out.  And those new curtains? Please.  I would also tell you to lock the door on the way out, but somehow that would be a bit too ironic.

The Worldwide Malware Signature Counter Lives On

At the bottom of the Triumfant home page is the Worldwide Malware Signature Counter, a fixture on the site since May of 2009.  The Counter was designed, according to the associated blog post marking its debut, “to graphically reinforce what many in the IT security industry believe is a growing problem that is being largely ignored – that the reliance on signatures to protect endpoints and servers against malicious attack is simply unsustainable”.  My only regret is that I never found a way to add the hard clunking sound from the timer on “24” to add emphasis.

I periodically check the Counter against reported malware counts to ensure that it is an accurate and fair representation of the signature story.  Truthfully, the Counter was designed to err on the side of understatement to avoid the impression of FUD or sensationalism, so I normally have to correct it up instead of down. Yes, IT security folks, there are actually marketing people with restraint.  Go figure.

Last week I updated the Counter to track to the signature counts reported by Symantec at the close of 2011.   Doing so led to a time of reflection on the genesis and objective of the Counter, and the changes in the threat landscape between then and now.

When Triumfant introduced the Counter three years ago, the world was still coming to terms with the evolution of malicious attacks and the hard realization that signature based protections could no longer be their primary shield. I would hope that there are very few serious members of the IT security community who need further convincing today.

Ironically, in the past three years the large vendors that owe their market presence largely on selling AV software have shifted their messaging.  Most dropped signature counts from their annual threat reports in spite of such counts being a featured staple in years past.  I noted in one blog post that one such vendor dropped any mention of the word “signature” completely.  In an interesting twist, some of these vendors now use the large malware sample numbers to sell other products and solutions in their portfolio.  The flood of annual reports that are the precursor for the RSA Conference scream numbers such as 75 million and 250 million for malware samples.  You have to feel for signature software: it made these vendors market leaders and it is now being dismissively kicked to the curb. Think Sunset Boulevard for security software.

Meanwhile, the battle to protect sensitive data and intellectual property continues to rapidly evolve.  The first malware sprung to life when sensitive information moved from corporate systems to the first personal computers.  Those early attacks now seem laughable against the volume and sophistication of the threats we face today, and things will only get more complicated when you consider the flood of mobile devices and BYOD machines that will soon be accessing corporate systems.  Furthermore, the adversary has changed from basement hackers to well organized, well funded, and highly motivated groups driven by monetary gain or political motives.  The sum total of this evolution creates a gap between signature based protections and the current reality that grows faster than a simple signature counter can capture.

The counter was a great visual to help people grasp the shift in the IT security world and helped bring attention to Triumfant’s ability to detect malware without signatures.  The counter often provoked people to ask if we were a replacement for signature based protections, and we always said no.  Signature based protections are a logical brick in the wall around IT assets, but they are just a brick, not the entire wall.  I should add that the Counter now serves as a symbol for all solutions that based their detection capability on some form of prior knowledge, not just AV.

My next thoughts went to the Counter itself and its continued existence on the Triumfant site.   After some consideration, I decided to keep it around because while the thinking of the IT security world has evolved there are still plenty of other business people outside of security that are still coming to terms with the concept.  Truth be told, I have an emotional fondness for the Counter and it is still a place for people to discover Triumfant and the uniqueness of our approach.

The Triumfant Worldwide Malware Signature Counter will live on.  Maybe I will finally add that sound effect.  Clunk…Clunk…Clunk…


Get every new post delivered to your Inbox.

Join 646 other followers