It's been a busy week for Apache Spark, the open-source computing platform that has built a lot of momentum. The Spark Summit took place this week in San Francisco and several vendors released Spark-related news.
First, Databricks, one of the primary vendors behind the continuing development of Spark, announced that its distribution of the open-source technology will be available on SAP HANA.
The idea is to put a single processing layer -- in this case, Spark -- at the center of an organization's data architecture. Spark proponents say it provides a simple interface for different storage systems, including Hadoop, NoSQL databases and relational databases. By partnering with SAP, Databricks wants to provide the connection between these data stores and applications that run on SAP HANA.
San Mateo, California-based Guavus also announced a new Spark integration in the latest release of its operational intelligence system Reflex. The new release has built-in integration with Spark, which will include pre-built analytic models for network operations, marketing, healthcare and security organizations.
The 1.0 release of Spark was officially introduced in June. Since then, development has continued. It offers a number of tools for data scientists, including the Machine Learning Library (MLlib), which has pre-built common machine learning algorithms. Coders are also working to create an interface for developers using the R programming language. Spark is currently included in every major distribution of Hadoop, including those from Cloudera, Hortonworks, IBM, MapR and Pivotol.
Business intelligence competency centers grow
The business intelligence (BI) competency center, with ultimate reporting to the IT department, is becoming an increasingly popular way to run BI programs, according to a new Wisdom of Crowds report from Dresner Advisory Services.
A common question businesses have when launching BI programs is which part of the organization should have ultimate responsibility. Obviously there is a large element of technology involved, which may suggest IT should own it. But the fact that it serves a particular line of business and works better when that business is closely involved argues in favor of business owning it.
The report found that many businesses are using the business intelligence competency center (BICC) model, which involves staff from IT and the line of business. In about one-third of these cases, IT has ultimate responsibility for the projects. Operations and finance were the next most common reporting departments at 20% and 18% respectively.
"For the past three years a growing percentage of our Wisdom of Crowds survey respondents have identified themselves as part of the BICC," Howard Dresner, founder and chief research officer of Dresner Advisory Services, said in a press release.
Spark helps merge cloud, database admin roles