Posts Tagged ‘columnar databases’

New data suggests that most of the industry’s data breaches could have been prevented by effective log data management, reporting and response. Consider the following from Verizon Business’ latest Data Breach Investigations Report, published just a few weeks ago:

• 86 percent of the victims had evidence of the breach in their log files, but most of them did not detect the breaches for months.
• 61 percent of the breaches were discovered not by the victim company, but by a third party outside the company.
• 96 percent of the breaches were avoidable through simple or intermediate controls.

This surprising data all points to one fundamental truth: Despite huge investments in security tools and expertise, many major businesses still don’t know when they have been breached. They are left trying to explain themselves when a breach that has been in place for months - or even years - is revealed to their top management by a third party.

Why don’t companies recognize breaches when they happen, especially when those compromises are readily evident in their log files?  Why can’t enterprises find the time to scan those logs for anomalies?  Most importantly, what can companies do to reduce the length of time between the initiation of an attack and its detection and remediation?

I see two fundamental reasons why most companies don’t have good answers to these questions. Both come up time and again from customers who come to SenSage having had problems with their old log management solutions. First, their log management solution can’t scale to meet their continuous event loading, storage and management requirements. There has been massive growth in the volumes of event data being generated in recent years - especially application-related events - so performance and scalability are becoming pressing challenges. And second, their log management solution does not support the sophisticated data analysis necessary to isolate the events that matter from the events that don’t. This is especially the case with stealthy “low and slow” attacks where you may need to analyze patterns from months or years worth of data.  Traditional log management systems with indexed searches of archives were never architected to deal with this.

If this all sounds painfully familiar, you should evaluate our unified SIEM and log management solution, purpose-built atop a clustered, columnar database. It scales from terabytes to petabytes while minimizing storage costs via patented compression algorithms and supports sophisticated query analysis through our console or your choice of business intelligence tools utilizing the industry’s only ODBC/JDBC interface to security event data.

permalink

In past blog posts I have often cited the need for a scalable event data warehousing capability to keep up with data collection and analysis requirements to address compliance and security operations. After hearing from dozens of customers about how they’re using SenSage to address their most critical security and compliance challenges, I’ve decided to focus less on event data warehousing and more on how our customers and partners are using SenSage. Towards the end of 2009, we searched for a way to net it out. In the end, it was pretty easy - Security Intelligence. This term sounds lofty at first, but once you learn how we think about it, I think you will find it very down to earth.

Of course, Security Intelligence is a variation of Business Intelligence or BI. BI solutions leverage the data management capabilities provided by data warehouses to deliver decision support information to business managers. Well, that’s exactly what Security Intelligence provides: essential decision support for security, risk management and compliance operations. Done right, Security Intelligence solutions are open, flexible, and scalable like traditional data warehouses while delivering deep security context.

Improved decision support is exactly what today’s security, risk management and compliance professionals are looking for. Detection and response to cyber-threats, regulatory compliance risks and investigating system failures all require thorough but simplified analysis of massive amounts of event data. Whether responding to an incident in real time or drilling through terabytes of related events to investigate the related context or improving a control, security professionals are asking for better decision support solutions.

As compared to Business Intelligence solutions, this is a bit of a niche play. These solutions are tailored to meet the needs of security, risk management and compliance professionals. But compared to the traditional SIEM and log management point products which are built on flat files, Oracle, or, worse, closed database management systems, Security Intelligence is a more flexible and sustainable approach.

SenSage is at the forefront of this technology, delivering Security Intelligence solutions that unify SIEM, log management and controls monitoring through a single analytics environment and data management architecture. Our customers are capturing the benefits of decision support in the security management context, leading to technology consolidation and process improvements not easily accomplished with the point products noted above.

We’ll be talking about Security Intelligence quite a bit in the coming months. Drop me a line, I’d love to hear your perspective.

permalink

I’ve been noticing a lot of discussion online about MapReduce and Hadoop recently. While MapReduce may seem new, implementations have been around for years. Let’s take a closer look.

MapReduce is a software framework introduced by Google to support distributed computing for large data sets on clusters of computers. The objective of MapReduce is to get extremely fast answers from massive amounts of data. In the “Map” step, the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker nodes process the smaller problem, and pass the answers back to its master node. In the “Reduce” step the master node then takes the answers to all the sub-problems and combines them to get the answer to the problem. One example of MapReduce is the Apache project Hadoop, a widely used open-source implementation of MapReduce.

So are these really new concepts? Not really. Some database systems with MPP architecture have been doing this for quite a while. While MapReduce is powerful, one of its drawbacks has been that each step of the MapReduce operation (filtering, grouping, and aggregation) is a separate, high-level programming abstraction that needs to be maintained by a developer and thereby increases data management total cost of ownership.

SenSage has been providing MapReduce capabilities with “in database” analytics commercially available since 2004. You might be saying, “yeah right”. Well, it’s true. We have over 400 deployed customers and patents to back it up.

We’ve simplified the promise of MapReduce. Namely, we’ve eliminated the hassle of intermediate programmatic effort to produce lightning-fast, in-memory analytics. SenSage combined a few pieces of our intellectual property with our MPP share nothing architecture to solve the problem:

• First, the SenSage columnar database supports parallel transformation and partitioning of data. In SenSage, SQL Map is like the group-by clause of an aggregate query. Reduce is analogous to the aggregate function (e.g., average or sum) that is computed over all the rows with the same group-by attribute.
• Second, since day one, SenSage has allowed users to write their own functions in SenSage SQL, which are automatically enabled for parallel execution using our MPP architecture. With Google, Hadoop, and many others, users have to write and maintain their own programs to accomplish the same thing.  With SenSage, users write standard SQL and SenSage does the rest.
• Third is “IntelliSchema” – this is where it gets really cool. This is a SenSage innovation that is an abstraction layer between the original data and the analysis tools, and enables our MapReduce engine to execute queries successfully even if the underlying data schema changes. Intellischema gives our customers the ability to handle a wide variety of data sources and write standardized libraries of analytics while still maintaining the fidelity of the original event data.  This allows any data source to automatically appear in relevant queries and reports.

It’s good to see technologies like MapReduce getting attention in the marketplace. As customers better understand the benefits, they can make more informed buying decisions.

permalink

Some people have described Hasso Plattner’s visionary speech at Sapphire earlier this month, “as the beginning of the end of the relational database as the mainstay of enterprise computing” (http://tinyurl.com/o8j3sz). In his keynote titled “The Power of Speed”, Plattner, SAP Chairman and co-founder, focused on the need for new software that enables business to move much faster and change the way work is done. He stated that companies today collect “unbelievable amounts of data,” (noting that the average SAP customer has seven to 10 years’ worth of data on disk) and that “how we digest that data is slow, and it’s getting slower because of the increased sizes of databases.”  Read More…

permalink