Saw an interesting column in InfoWorld on “Learn to Love Your Log Files� -  http://tinyurl.com/lvhabg

The author, Roger Grimes, highlights a theme that is increasingly getting increased attention – the value of log files.  The article gives practical ideas for implementing and managing log management systems.  He also provides an interesting perspective on how SIEM and log management technologies fit together.

In my opinion, SIEM originated with the vision to be the single-pane of glass – to separate the signal from noise.  From an architectural perspective, data management was generally an afterthought.  Events were normalized and data discarded after a few weeks.  As a result, the initial wave of vendors built their solutions around familiar data management systems such as Oracle databases or flat files.  Over time, the reporting requirements became more demanding and the amount of data to be analyzed increased significantly.

The pendulum has shifted – data management is a central buying criterion for a logging or SIEM solution. Compliance might have started this trend, but now security is giving it the next push.  Why?  Threats are more sophisticated.  Insiders don’t generate failed logons.  So, you need to keep months of valid session detail if you want to find the low and slow anomalies.  In order to keep up with these demands, many customers are expanding their data retention period as well as the scope of data analyzed to include ERP applications, credit card and ATM transactions….their most sensitive data.

The implication of these trends is massive data stores and more sophisticated data analysis – even for small firms.  Log data repositories can easily reach into the 10s of terabytes for small firms and hundreds of terabytes for larger firms.  It’s no surprise that for many organizations, security and event data is their largest single data store.  As a result, customers are looking at long-term ROI and are pulling their enterprise data warehouse architects into data governance & compliance efforts.

Today, people from diverse roles across the enterprise need immediate access to security and GRC information.   Having said that, you can’t trade off accuracy and completeness for ease of use, and it has to be tamper-proof.  The implication is you need a system that is easy to use AND provides reports and trending information that is 100% accurate.  That’s why some vendors who claim to be “Google for Logs� (fast but not 100% accurate) will have difficulty addressing the reporting, forensic, and retention requirements of the log management market.

Check out the article - another good contribution to the conversation.

permalink