Security Configuration Management – Don’t Fall for the Old Saw of Patch Management

April 16, 2010

Yesterday I attended a customer event for one of the larger IT security firms and one of our partners.  During one of the sessions, another partner gave a presentation on security configuration management that nearly drove me to a Kanye West “grab the microphone” moment.

The partner in question regaled the attendees with a story of automated configuration management that involved long intrusive scans, followed by analysis to identify problems, followed by the issuance of what are essentially patches to correct non-compliant machines.  This process seemed horribly cumbersome and certainly did not meet my definition of automated.  And the remediations for the detected problems were pulled from a list of pre-written remediations in a one-size-fits-all approach.

Worse of all, the process happened infrequently – monthly at best, perhaps quarterly or twice yearly.

The saw blades are a visual of configuration drift over time. The length of the saw teeth represents the amount of drift and the size of the gap between them represents the time between corrections.  If the height of each saw tooth indicates how much configuration drift you will experience with large gaps between configuration corrections, which do you think represents the most secure environment?  The bigger the teeth, the higher the organizational risk.  You want the hacksaw blade.

My negative reaction comes from knowing there is a better way to deliver security configuration management Triumfant will continuously scan for changes to endpoint machines and detect when the machine is in a non-compliant state.  Triumfant’s analytics will evaluate the changes to the machine, create a remediation for that problem, and return the machine to compliance.  Remediations are created on the fly to address specific detected problems on each machine.  The remediations are surgical, contextual, and situational.  The remediation is delivered to the machine and executed by the agent.  All of this can be set to a one-touch confirm from the administrative console or fully automated.  And we will open a touble ticket, populate the ticket, and close it to track the process.

The result – your organization starts every day in an audit ready state.  The maximum drift is 23 hours and 59 minutes.  Not a month, 3 months, or 6 months.  No need for heavy, obtrusive scans, no human intervention needed to write remediation scripts, no additional patching activity.

Folks, patching is a big part of the problem, so why would you get excited about any so-called solution that is essentially a patch management process?  Patching is hard and rarely done well.  Why do you think there is so large a time gap between correction cycles with this technique?  Take a hard look at many of the companies pushing configuration management – they often have their roots in patch management and that is how they address the problem.

Security configuration management effectively reduces the attack surface on each machine, but this is achieved only when the configurations are continuously enforced.  When you can detect and remediate problems every day, you create what we call persistent security readiness.  Don’t settle for old school techniques and large gaps between corrections because monthly or quarterly is not persistent.  There is a better way.


A Condensed Guide to the Security Fails of 2009

December 23, 2009

The past several weeks I have been posting a series I called the Security Fails of 2009.  It was designed to be a look at stories that illustrated the challenges faced in IT security as well as some of the broader issues shaping the industry. 

For your convenience, here is a recap with links:

12/10 – The Marine One Breach – illustrates the threats created by unauthorized applications.

12/14 – The Strange Case of the Missing Cyber Czar – a look at the seven months that had passed since the announcement of the position in May.  Obviously the position has been subsequently filled.  Coincidence?

12/16 – Conficker Becomes a Media Darling.

12/18 – Adobe Takes the Exploit Crown from Microsoft.

12/21 – The Heartland Payment Systems Breach – Lessons learned form the largest breach of customer data to-date.


Security Fails of 2009 – The Heartland Payment Systems Breach

December 21, 2009

This is the fifth in the series of Security Fails of 2009.  As 2009 draws to a close I think no one would argue that this has been an extremely eventful year for IT security.  While others will soon be trotting out their “best of 2009” lists, I thought I would instead visit some of the prominent fails of 2009. 

In January of 2009, it was disclosed that Heartland Payment Systems had experienced an intrusion into their computers that may have compromised over 100 million customer records.  After the dust settled, the breach was found to involve 130 million customer records, pushing this breach well past the previous record represented by the 2007 TJX breach that compromised 94 million records.  Heartland processes 100 million payment card transactions per month for 175,000 merchants.

By December the attack was traced to admitted TJX intruder Albert Gonzalez who eventually entered into a plea agreement on the Heartland breach and additional charges that he hacked into Hannaford Brothers, 7-Eleven and two other unnamed national retailers.  Heartland has allocated $12.6M for the clean-up, and as of today Heartland was still settling with American Express ($3.6M) and resolving other class action suits.

The scope of the breach re-energized conversations about the efficacy of the PCI standards and the general state of fraud protection for card based transactions.  The dialogue became more interesting when Heartland CEO Robert Carr did an interview with Bill Brenner of CSO Magazine where Carr laid the blame squarely on the audits done by their Qualified Security Assessors (QSAs).  Carr’s comments were viewed by many in the security community as “disingenuous” as most believe that the source of the breach could have been eliminated if Heartland had applied some generally accepted security controls. 

PCI has long been an industry hot button, and the Heartland attack was illustrative of the issues at hand.  Heartland appeared to be in full compliance with the PCI standards, but was attacked by essentially a “garden variety” SQL injection.  In an interesting twist, Heartland’s traditional signature based tools missed the attack, but the attackers actually used antivirus software to cover their tracks and avoid detection. 

So what are the lessons learned?  Heartland demonstrates that even the most sophisticated companies in regards to IT security are still far too reliant on signature based tools and must look to new and evolved technologies to close security gaps that allow long known vectors such as SQL injection to breach their perimeters.  Heartland is also a great “exhibit A” that compliance does not equal security; it is only a temporary measure that certain standards were in place at a point in time.  Finally, in spite of calls to action to rid the card processing industry of fraud, there is not much evidence that anything other than rhetoric came from the attack, so we can fully expect to see another Heartland in 2010.


Why Bad Things Happen to Good Endpoints

November 12, 2009

I was with a prospect the other day and was asked what, at least for me, was a very thought provoking question.  We were discussing the two major areas of application for Triumfant – continuous enforcement of security configurations and real-time malware detection and remediation – and he asked why you would need the latter if the former was done properly.  In other words, if all of my endpoint protections are in place and everything is properly configured, why am I still getting attacked?

Simple and logical question, right?  But it led me to think long and hard why attacks happen at a very elemental level.  We in security face this question from the powers that be because they cannot understand that attacks still come even though we have added multiple layers of defense. 

After consideration I came up with three reasons.  For perspective, my reasons are very much endpoint centric and presume the attacks have already made their way through protections on the network level, so this is not a cloud to ground holistic view.  Each reason is based on the assumption that the preceding reason(s) have been fully addressed and the represented gap is closed – each reason stands on its own as a problem.  And I will resist the urge to plug in how Triumfant addresses each gap, but I have noted blog entries that do if you care to read on.

Here are my three reasons:

  1. Attacks get through because the machines and the protection software deployed to protect them are not configured to be secure.  The analogy is simple: the most well designed and secure deadbolt lock only secures a door when the deadbolt is engaged.  Too frequently, endpoint protection tools are either improperly installed or improperly configured to perform the tasks for which they are intended, so attacks make it through.  For how Triumfant addresses the configuration management gap see “A New Approach to Configuration Management”.
  2. Attacks get through because traditional endpoint protection tools miss known attacks even when there is a signature for that attack and the protection is properly configured.  The failure rate depends on whose statistics you chose to use, but Gartner puts the detection failure rate at two to ten percent while other studies show failure rates exceeding fifty percent.  Given there will be well over 5M signatures by the end of 2009, ten percent is non-trivial.  See “Antivirus Detection Rates – It is Clear You Need a Plan B”.
  3. Attacks get through because they have been carefully designed and engineered to evade traditional endpoint protections.  These include zero day attacks, rootkits, targeted attacks and the work of the maliciously intended insider.  Zero day attacks are more generic in nature and broker on the fact that most tools require prior knowledge to detect an attack.  Targeted attacks are designed to specifically infiltrate networks and applications to retrieve sensitive information or financial data.  See “It is Raining and You Will Get Wet”.

I am not saying this is groundbreaking thinking here, but if you put things into this perspective, it clearly defines the gaps in protection and subsequently provides a roadmap of what must be done to protect your endpoints.  Reducing the attack surface is clearly not enough.  Antivirus is not getting it done – even the AV vendors say so.  And the bad guys are relentless in their pursuit to exploit any crack in the defenses. 

So what do you think? Too simple or simply brilliant?


A New Approach to Security Configuration Management

August 31, 2009

In my last post  called If We Know How Breaches Happen, Then Why Aren’t We Doing Something?, I addressed a recent post by Rich Mogull in the Securosis blog called “We Know How Breaches Happen” where Mogull rightfully asked if we know that security configuration problems are at the root of a significant number of breaches, then why more isn’t being done to enforce essential security configuration practices. 

In my post I ended by noting that a better method of automating enforcement of security configurations was needed as most security configuration management tools on the market today are a derivation of patch management techniques and processes.  They rely on the application of pre-written scripts for remediation that are pushed out like patches.  Some tools literally have tens of thousands of these pre-written remediations.   If a new problem is detected, someone – either internally or the vendor – has to write a new script which is then pushed out.  This effort can be non-trivial as new scripts must be handled the same way that a new patch must be tested and managed.  The application of these scripts is very brute force and non-specific.  If someone gets sick (one computer) then everyone has to take the pill (the script). 

That is why an alternative is required.  patch management, or the lack thereof, is constantly cited as a contributing problem to security and specifically, security configuration management.  If security configuration management tools rely on what essentially is the patch management process, why would we expect those solutions to solve the problem? 

Triumfant Resolution Manager takes a decidedly different approach to security configuration management.  It continuously enforces security configurations and policies by scanning every computer, every day, detecting when a machine is non-compliant, building a remediation on the fly for the specific problem for each specific machine and applying these remediations to return non-compliant machines to compliance.   There are no scripts to write, no scripts to be pushed out to every machine.  Triumfant has automated the process of detection, analysis and remediation.  You start every day with every machine audit ready and at the highest state of security readiness.

So how does Triumfant learn the desired security configurations to enforce?  The first way is that it learns the rules from the environment by taking thousands of attributes from every machine and correlating them into the rules that compose what we call the Adaptive Reference Model.  The model is a normative baseline of all of the correlated attributes across the endpoint environment expressed as rules for specific configuration settings.  Essentially, Resolution Manager will learn what is normal and then enforce that normal.  For more insight into the grouping and correlation performed by the Adaptive Reference Model please see this previous post.

Obviously many environments are in a state where there is no satisfactory normal.   The learned option still works, but requires some finesse and is dependent on having a set of machines that can serve as a golden image of your desired configuration.  In this case, you would simply start by pointing Resolution Manager at these machines first, allowing the Adaptive Reference Model to be built with the rules learned from these machines.  Then as you add other machines to the model, they will be assessed against the rules learned from the golden image machines and remediated to be brought into compliance with those rules.    

Seldom is the world as tidy as a golden image.  There are the “except fors” and special cases that keep IT security teams busy.  So the second way to establish security configurations is through explicit rules and policies that are created within a simple wizard driven interface.  These can apply to all machines in the endpoint population or to specific groups, making it simple to accommodate the requirements of groups or job functions.   Rules are extremely flexible and can be extremely precise and specific or employ generic filters to cover broader subject matter.  Using this approach, organizations can deploy security configurations that are both practical and appropriate.

The beauty of Triumfant Resolution Manager is that you can employ a hybrid model.  You can explicitly state the configuration rules that are most important, and Resolution Manager will extend the explicit rules with learned rules to cover the rest.  No other tool can provide this two phase approach, effectively extending the reach of security configuration management well beyond that which is explicitly defined.   

The ability of Triumfant Resolution Manager to continuously and automatically enforce security configurations creates enormous benefit for our customers.  This capability provides all of the advantages of security configuration management with less human interaction and extends the capability far beyond the explicit configuration rules.  The end result is lower risk and increased security readiness.   While it is the ability of Resolution Manager to detect the malicious attacks that evade other endpoint security software certainly garners a lot of attention, the ability to ensure that the endpoint population begins every day properly configured is equally critical in securing any organization.


If We Know How Breaches Happen, Then Why Aren’t We Doing Something?

August 28, 2009

In his August 26 blog post on the Securosis Blog called “We Know How Breaches Happen”, Rich Mogull made some very good observations about the cause of data breaches. According to Mogull:

“If we look across all our sources, we see a consistent picture emerging. The vast majority of cybercrime still seems to take advantage of known vulnerabilities that are can be addressed using common practices. The Verizon report certainly calls out unpatched systems, configuration errors, and default passwords as the most common breach sources.”

When I was with Cybertrust, Peter Tippett, one of the early pioneers in antivirus software now with Verizon Business, would make the impassioned case (he still is) that following a relatively small number of essential practices would lower an organization’s risk significantly.  Tippett’s researchers from Cybertrust are at the core of the Verizon team that publish the Verizon Data Breach Investigations Report which notes that “2008 continued a downward trend in attacks that exploit patchable vulnerabilities versus those that exploit configuration weaknesses or functionality.”

In other words, a little discipline in security configuration management would go a long way toward making organizations more secure and eliminating the low hanging fruit used by hackers.  Hackers are people too, and like the rest of us they will take the path of least resistance.  They could choose the difficult path of engineering a new zero day attack. Or they could take the far more simple approach of using an exploit that leverages a common misconfiguration known to exist in a significant number of endpoint machines and build an attack with enough variation to evade the signatures in place for earlier versions of that attack. 

GGGR1You can almost picture a Glengarry Glen Ross hacker’s boiler room full of hackers under quota and a boss telling the crew that “Coffee is for hackers that Hack!”  It is just too simple to spin up an exploit that picks off the multitude of unpatched and misconfigured endpoints.

In a separate post, Mogull talks about the Data Breach Triangle in the context of fire triangle (oxygen, heat, fuel – take away any one of the three and the fire goes out): the sides of the triangle are the three components needed for a breach to occur, so removing any one side prevents the breach.  Good security configuration management should help eliminate the triangle side Mogull calls the exploit, which is the vulnerability or flaw that provides the hacker the path to the data.

Given the obvious linkage between breaches and configuration problems, why hasn’t security configuration management become an essential component of endpoint security strategies?  The answer is steeped in irony given that configuration management has replaced patchable vulnerabilities as the exploit of choice.  Many of the companies that offer security configuration management use what are essentially patch management tools as the technical implementation.  Think of the number of patch management vendors that now offer security configuration management. These solutions essentially push out configurations and remediations the same way that they would push out a patch. 

Why is his ironic? While this is a proven and technically sound approach, patching is somewhat universally recognized as problematic.  It is therefore sensible and logical to ask why we would expect that using the same type of tool and underlying processes would prove to be successful in addressing configuration management.  Patching tends to be a brute force process, while configuration management requires much more flexibility and finesse.  And creating a patch (call it any name the vendor gives it – it is still a patch) is a human resource intensive process that requires someone to write, test and deploy a script.  

It would be fair of the patch management vendors to note that a lack of institutional commitment and sound process are also contributing factors and I would agree.  But the parallel between patch management and configuration management is too obvious to ignore.  Mogull’s observations are backed by numerous studies and analysis that cite unpatched systems and security configurations errors as a problem, so the evidence would indicate that these tools are not getting the job done. 

Clearly a different and more automated approach is appropriate, and in my next post I will tell you how Triumfant addresses this problem.  Specifically, I will detail how Triumfant continuously enforces security configurations by detecting machines that are out of compliance with desired configurations and automatically build a remediation to return the machine to compliance.

What is clear is that we need to address these foundational problems to become more secure and reduce organizational risk.  Perhaps now the tools have caught up with the problem and we can make the road of that hacker a bit more difficult.


Exhibit A for Bad Advice – A Questionable Recommendation from the New York Times

May 15, 2009

Yesterday a friend sent me an article in the New York Times asking my opinion on a recommendation made by the author regarding improving performance on home PCs.  In the article Five Controversial Ways to Speed Your PC, author Paul Boutin recommends that users “uninstall your antivirus software” because he perceives the threats are an overhyped and basically scaremongering by his fellow journalists. 

I hope the writer has the guts to come back and tell his readers just how long his machine survived unprotected.  I have seen studies where unprotected PCs have been connected to the Internet and are infected in minutes and part of a botnet in hours.  In my opinion, this recommendation was irresponsible and could cause a lot of people to lose personal data on their home machines. 

But this is just the kind of behavior that I pointed out in my recent post about “Stopping Stupid”.  All of the security software, policies and configurations cannot protect against the human element, especially when it looks to do something like the recommendation for this NY Times article.  Because you know that there are people in the workplace that read the article, decided that their AV software was the reason their machines at work were not as fast as they want, and started the process of disabling or eliminating their AV software on their work PC.  If this were an old horror movie, CISOs and IT techs would be an angry mob on their way to Mr. Boutin’s office with torches and pitchforks. 

That is why security configuration management tools have got to be more than a one-way push of configurations to ensure endpoint security.  These products must have every machine, every day vigilance to verify that the configurations and policies are in place and take the steps to remediate the machines if they are not.  The only way to fight incompetence or ignorance is through relentless repetition.   And since stupid is a free-style art form, signature based tools and pre-written remediation scripts will not get the job done.  The security configuration management tool has to be able to do situational remediation to address problems as they are encountered.

Lots of endpoint protection and configuration management tools may say they do exactly that, but they don’t.  They are pushing scripts.  I suggest you ask for more from your security configuration management tool and make sure you choose one that will stand against the crafty work of the maliciously intended cyber criminal as well as stand in the gap against user incompetence and ignorance.


Stopping Stupid – Dulling the Edge of Hanlon’s Razor

May 12, 2009

There is a corollary to Murphy’s Law called Hanlon’s Razor that goes as follows:

“Never attribute to malice that which can be adequately explained by stupidity, but don’t rule out malice.”

In the world of IT security, much risk and ultimately damage is caused by stupid in the form of ignorance or selfishness or just plain zero brainwave activity.  Because nothing can render defenses useless faster than human stupidity. 

So how do you stop stupid?  It is not easy, because a quote by Friedrich Schiller says:

Against stupidity, the gods themselves contend in vain.” 

What is needed is something that is doggedly persistent and tireless in its defense against stupid.  Something that never throws up its hands in the face of relentlessly repetitive stupid.  Something that no matter how many times it must turn stupid away will do so with a singular purpose.   

Triumfant resolution Manager does a great job of security configuration management.  It will continuously enforce security policies and configurations, and when it sees non compliance it will automatically create a remediation to return the endpoint machine to compliance.  It will also detect machines that have been changed in such a way that is anomalous to other like machines in the endpoint population, and based on how anomalous the change is, either create a remediation or alert the administrator. 

In other words, Triumfant will stand tirelessly, continuously, and relentlessly against stupid.  Every time a user sets his or her machine to a configuration or state that would create a vulnerability, Triumfant will set it back.  If the user then changes the setting the next day, Triumfant will set it back.  If the user disables their antivirus agent, Triumfant restores it.  

No other tool that I know of is equipped to address the human element of security at the endpoint like Triumfant.  The ability to continuously scan a machine and build a remediation on the fly is completely unique in the market and is uniquely capable to mitigate the effects of stupid.  Given that there is no human intervention needed to remove the effects of stupid, your organization gets a solution that delivers with near zero human costs. 

A loosely attributed quote from Einstein summed up stupid as follows:

“Only two things are infinite, the universe and human stupidity, and I’m not sure about the universe.”

But combining Triumfant’s configuration management capabilities to Triumfant’s ability to detect, analyze, and remediate a malicious attack without signatures and without human intervention, and you have a really powerful tool to add to your security strategy.  It won’t completely mitigate stupid, but it will win one small skirmish in the war and dull the edge of Hanlon’s Razor.


The UC Berkeley Breach – You Don’t Know What You Don’t Know

May 11, 2009

You don’t know what you don’t know. 

Or, you don’t know what you are content to not know.  Or, you don’t know what you don’t know because you have put way too much trust in one vendor, one product, or one methodology.

No matter what fits you and your organization the best, there is a very good chance that you are under attack right now and you do not know it.  Data is being surgically extracted from your systems on a continuous basis.

Ask the folks at UC Berkeley who are coming to terms that a system containing medical information and personally identifiable data was breached for over six months.  The estimate is that data was stolen for over 160,000 students and associated people.  The capper for me was that the breach was only discovered during routine maintenance when messages left by the hackers were found.  Had the hackers not left these messages, it is highly likely that the breach would still be active.

The UC Berkeley story is not unique; it is just the latest of what has been discovered.  While at Cybertrust our Forensics/Incident Response team would regale me with stories of breaches that went undetected for what seemed to be unbelievably long periods of time.  But in the face of these stories, organizations continue to reconcile the threats away as long as it does not happen to them.  Or more accurately, as long as they don’t know it is happening to them. 

I have used this quote before, but it is apropos: Churchill said that man occasionally stumbles over the truth, but far too often picks himself up and continues on as if nothing happened.  UC Berkeley is certainly not the first such incident we have stumbled on, and certainly it won’t be the last.  But maybe you may not want to pick yourself up and continue on so quickly and consider that you don’t know what you don’t know. 

And then consider if there are alternative technologies that can help you see more clearly what you don’t know.   Look past the “usual suspects” and consider that there are viable alternatives in the market that may bear a close look.  We happened to think what we offer at Triumfant is one of those technologies.  But regardless of what you evaluate, you need to consider if you are really as protected as you think.

Because when you find out what you don’t know, it may be on the front page of the morning paper.


Winning the Fight Against Unauthorized Applications

May 5, 2009

I read an article in CIO Magazine about businesses losing the fight against employee applications.  The gist of the article is the loss of bandwidth and the other distractions caused by user installed programs and things like YouTube.  The story cited a survey by Palo Alto networks that found that 82% of the businesses surveyed had an average of six peer-to-peer applications.

None of that surprised me, as our experience is that organizations have easily ten times the number of applications running on endpoint machines that even their worse estimates.  Triumfant Resolution Manager does a stellar job of cataloguing all of the applications running on the endpoints, and that process always leads to lively discussions and discovery.  One customer swore that they only had 150-200 applications running on their endpoints, and we found over 9,000 unique applications in a population of 6,000 endpoint machines.

Unauthorized applications have always been the bane of IT support teams, as the introduction of new applications may cause conflicts that detrimentally affect system performance or create conflicts with other business applications that result in outages.  Having to manage these problems translates to a real and significant expense for the organization.  Recently, applications that are based on peer-to-peer communications have been shown to be the source of vulnerabilities and have been the direct cause of data breaches such as the leakage of the Marine One Helicopter plans.  It is clear that unauthorized applications create unnecessary expense and risk for the organization.

What does surprise me is that the story was about the problem being fixed by advanced firewall capabilities.  Granted I am no expert on firewalls, but how is a firewall going to eliminate unauthorized applications?  Managing unauthorized applications comes down to two inseparable things: sound policies and a tool to continuously enforce those policies.

First, organizations have to come to terms with personal use policies and the growing presumption that use of a personal computer means that it is the employee’s personal machine for their personal use.  It is a given that if an organization does not have personal use guidelines, employees will load anything and everything on their endpoint machines.  Particularly if everyone has Administrator access to their machine which is another whole topic of discussion.  So unless you have a set of personal use policies – install authorities, a whitelist of acceptable applications, zero tolerance of peer-to-peer applications – that are well defined and have some teeth, this problem will be yours forever.

I find the whole personal business on a business machine to be perplexing.  I have my own laptop that I use for my personal business.  My music, my personal email, and any other personal applications are on this laptop.  If I want to check my personal mail at lunch, I bring this machine to the office.  I do not want my personal business on a company machine as much as the company does not want my personal business on their machine.  Is it a pain to carry two laptops?  You bet. B ut that, as they say, is how I roll.  But I know I am an exception and many now come to consider the laptop handed to them by work as their personal playground.  So cranking up some personal use policies may be seen as a “take-back” to the employee base, but you will have to stop the tide some timeas there is simply too much risk to the organization to do nothing.

Second, you need a tool to enforce the policies.  There are many whitelist/blacklist tools on the market that will manage what applications can be installed on a given machine.  Triumfant does a great job of managing applications on endpoint machines, and we have a customer success story on our web site where we manage applications on 12,000 endpoints for the Pentagon by the U.S. Army Information Management Support Center (IMCEN).  Triumfant detects and removes unauthorized applications, and policies can be tuned by work group down to the individual PC level to accommodate exceptional cases and specific working requirements for different teams.  For example, it may be policy to eliminate Skype from all machines except for those endpoints used by the teams that do extensive international travel. 

Having the right policies in place and the right tool to enforce those policies can make the task of controlling unauthorized applications much simpler and far less expensive than handling the problem reactively.  IMCEN tells us that Triumfant saves them $8 per machine per month in human costs of managing unauthorized software.  It is possible to effectively manage this problem and save human resources!  Best of all, organizations can significantly reduce IT security risk by eliminating these unauthorized programs.