Infosec pros have been hoping for some time that big data and analytics can be harnessed to improve cyber security. A fledgling open source project with genes from Intel and Cloudera is another step in that direction.
Originally called the Open Network Insight project for analyzing flow and packet data, the two companies have donated their work to the Apache foundation incubator, where it is now called Apache Spot. The move was announced last week at Strata+Hadoop World.
The goal of Spot is to focus on “hard security problems” – detecting events such as lateral movement, side-channel data escapes, insider issues, or stealthy behavior in general, says a Github wiki. “Spot can be deployed incrementally to realize immediate ROI, but is also meant to support an organization’s growth and maturity to achieve complete threat visibility as part of its protection strategy.”
It is hoped organizations will adopt Spot and spend more time building the analytics and visualizations that help discover cybercrime and less time building systems to ingest, integrate, store, and process any volume or variety of security data.
According to Computerworld U.S., Spot is based on Cloudera’s big data platform, which uses Apache Hadoop, for infinite log management and data storage scale along with Apache Spark for machine learning and near real-time anomaly detection. The software can analyze billions of events.
For example, the wiki notes, NetFlow can be used to analyze IP traffic information across corporate networks. However, in a large environment that could amount to billions of NetFlow events per day. Spot can handle that. It can integrate many different data sources in a data lake then add operational context to the data by linking configuration, say its supporters, to deliver risk-prioritized, actionable insights.
“The idea is, let’s create a common data model that any application developer can take advantage of to bring new analytic capabilities to bear on cybersecurity problems,” Mike Olson, Cloudera co-founder and chief strategy officer, was quoted by Computerworld as telling the conference.
In addition to Intel and Cloudera, other companies that have been working on the project include Anomoli, Centrify, Cloudwick, Cybraics, eBay, Endgame, Jask, Streamsets and Webroot. Cloudera announced version 1.0 of what was then called Open Network Insight in February.
In a news release last week announcing the donation to Apache, Cloudera said Spot provides common open data models for network, endpoint, and user. These Open Data Models provide a standard format of enriched event data that makes it easier to integrate cross application data to gain complete enterprise visibility and develop net new analytic functionality. And those models will allow organizations to share analytics with each other.
The out of the box machine learning capabilities will allow organizations to quickly discover abnormal and malicious behaviors using Apache Spark, it said. In addition, organizations to run analytics against comprehensive historic data sets, helping organizations identify past threats that have slipped through the cracks.