Posts Tagged ‘columnar databases’

In past blog posts I have often cited the need for a scalable event data warehousing capability to keep up with data collection and analysis requirements to address compliance and security operations. After hearing from dozens of customers about how they’re using SenSage to address their most critical security and compliance challenges, I’ve decided to focus less on event data warehousing and more on how our customers and partners are using SenSage. Towards the end of 2009, we searched for a way to net it out. In the end, it was pretty easy - Security Intelligence. This term sounds lofty at first, but once you learn how we think about it, I think you will find it very down to earth.

Of course, Security Intelligence is a variation of Business Intelligence or BI. BI solutions leverage the data management capabilities provided by data warehouses to deliver decision support information to business managers. Well, that’s exactly what Security Intelligence provides: essential decision support for security, risk management and compliance operations. Done right, Security Intelligence solutions are open, flexible, and scalable like traditional data warehouses while delivering deep security context.

Improved decision support is exactly what today’s security, risk management and compliance professionals are looking for. Detection and response to cyber-threats, regulatory compliance risks and investigating system failures all require thorough but simplified analysis of massive amounts of event data. Whether responding to an incident in real time or drilling through terabytes of related events to investigate the related context or improving a control, security professionals are asking for better decision support solutions.

As compared to Business Intelligence solutions, this is a bit of a niche play. These solutions are tailored to meet the needs of security, risk management and compliance professionals. But compared to the traditional SIEM and log management point products which are built on flat files, Oracle, or, worse, closed database management systems, Security Intelligence is a more flexible and sustainable approach.

SenSage is at the forefront of this technology, delivering Security Intelligence solutions that unify SIEM, log management and controls monitoring through a single analytics environment and data management architecture. Our customers are capturing the benefits of decision support in the security management context, leading to technology consolidation and process improvements not easily accomplished with the point products noted above.

We’ll be talking about Security Intelligence quite a bit in the coming months. Drop me a line, I’d love to hear your perspective.

permalink

I’ve been noticing a lot of discussion online about MapReduce and Hadoop recently. While MapReduce may seem new, implementations have been around for years. Let’s take a closer look.

MapReduce is a software framework introduced by Google to support distributed computing for large data sets on clusters of computers. The objective of MapReduce is to get extremely fast answers from massive amounts of data. In the “Map” step, the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. A worker node may do this again in turn, leading to a multi-level tree structure. The worker nodes process the smaller problem, and pass the answers back to its master node. In the “Reduce” step the master node then takes the answers to all the sub-problems and combines them to get the answer to the problem. One example of MapReduce is the Apache project Hadoop, a widely used open-source implementation of MapReduce.

So are these really new concepts? Not really. Some database systems with MPP architecture have been doing this for quite a while. While MapReduce is powerful, one of its drawbacks has been that each step of the MapReduce operation (filtering, grouping, and aggregation) is a separate, high-level programming abstraction that needs to be maintained by a developer and thereby increases data management total cost of ownership.

SenSage has been providing MapReduce capabilities with “in database” analytics commercially available since 2004. You might be saying, “yeah right”. Well, it’s true. We have over 400 deployed customers and patents to back it up.

We’ve simplified the promise of MapReduce. Namely, we’ve eliminated the hassle of intermediate programmatic effort to produce lightning-fast, in-memory analytics. SenSage combined a few pieces of our intellectual property with our MPP share nothing architecture to solve the problem:

• First, the SenSage columnar database supports parallel transformation and partitioning of data. In SenSage, SQL Map is like the group-by clause of an aggregate query. Reduce is analogous to the aggregate function (e.g., average or sum) that is computed over all the rows with the same group-by attribute.
• Second, since day one, SenSage has allowed users to write their own functions in SenSage SQL, which are automatically enabled for parallel execution using our MPP architecture. With Google, Hadoop, and many others, users have to write and maintain their own programs to accomplish the same thing.  With SenSage, users write standard SQL and SenSage does the rest.
• Third is “IntelliSchema” – this is where it gets really cool. This is a SenSage innovation that is an abstraction layer between the original data and the analysis tools, and enables our MapReduce engine to execute queries successfully even if the underlying data schema changes. Intellischema gives our customers the ability to handle a wide variety of data sources and write standardized libraries of analytics while still maintaining the fidelity of the original event data.  This allows any data source to automatically appear in relevant queries and reports.

It’s good to see technologies like MapReduce getting attention in the marketplace. As customers better understand the benefits, they can make more informed buying decisions.

permalink

Some people have described Hasso Plattner’s visionary speech at Sapphire earlier this month, “as the beginning of the end of the relational database as the mainstay of enterprise computing” (http://tinyurl.com/o8j3sz). In his keynote titled “The Power of Speed”, Plattner, SAP Chairman and co-founder, focused on the need for new software that enables business to move much faster and change the way work is done. He stated that companies today collect “unbelievable amounts of data,” (noting that the average SAP customer has seven to 10 years’ worth of data on disk) and that “how we digest that data is slow, and it’s getting slower because of the increased sizes of databases.”  Read More…

permalink