Business intelligence (BI) projects remain a priority for businesses, with emerging areas like predictive analytics gaining traction as companies look to exploit their burgeoning data resources to better extrapolate trends, improve product quality and create competitive advantage.
Traditional BI products, including dashboards and ad hoc query and reporting tools, leverage historical and, in some cases, real-time data to identify trends and answer questions related to what happened or why it might have happened. Predictive analytics, on the other hand, churns through large volumes of both historical and real-time information, building a model that can be used to project what might happen.
Despite the crunch on IT budgets, businesses are willing to spend on BI and predictive analytics. According to International Data Corp.'s Worldwide Business Intelligence Tools 2007 Vendor Shares report, the total BI market grew 12.1% to reach $7.05 billion in 2007, its third straight year of 12% growth. Advanced analytics tools were a standout segment, growing at a slightly faster pace of 13.1% to reach nearly $1.4 billion in 2007.
Predictive analytics is business intelligence's future
While the numbers support the idea of an expanding market, this so-called crystal ball capability isn't really new. Predictive analytics tools from established vendors like SPSS and SAS Institute have been around for decades, as well as more recently from niche players such as FICO and KXEN. Yet the category never really achieved widespread adoption, largely because of the highly specialized nature of the practice and the usability of the tools, which are considered too arcane and complex by the bulk of mainstream users. As a result, predictive analytics tools have been embraced mostly by highly trained experts and statisticians in industries such as finance as a way to spot fraudulent activity and evaluate potential credit risks; or in the retail sector, as a means for predicting cross-sell and up-sell opportunities among an existing customer base. The telecommunications sector has been another early adopter of predictive analytics, using the tools to predict likely customer activity such as churn and/or payment rates.
Given the harsh economic times and highly competitive nature of today's business climate, nontraditional sectors like manufacturing are also starting to see potential for such advanced analytical capabilities. Industry experts suggest that companies consider investing in predictive analytics tools if they have a repeatable process that generates a lot of data and for which a slight improvement in execution could save money or generate additional revenue. Manufacturers, for example, might leverage the technology to get a sense of mean time between failure (MTBF) rates for a variety of processes or designs, aiding their efforts to boost quality. Forecasting supply chain bottlenecks and materials management to predict the reliability of certain materials are other potential use scenarios.
"What's new is that businesses are finally figuring out that they need these capabilities, and they're seeing some practical examples of how they can use these tools in everyday life," said Claudia Imhoff, president of data warehouse consultancy Intelligent Solutions.
BI vendors bolster their advanced analytics capabilities
In response to this blossoming customer demand, BI vendors are upping the ante, aggressively rounding out their suites with predictive analytics and other advanced analytics capabilities. Some, like SAP, have partnered with established leaders in this field; others -- IBM, for example -- have made key acquisitions ( IBM acquired SPSS in July 2009). While the BI suites have been expanded to include capabilities, there is still work to be done in terms of full-blown integration.
"It always takes a couple of years," said Wayne Eckerson, director of TDWI Research, a data warehouse and business intelligence consultancy. "Slideware comes immediately, followed by veneer interfaces, then a series of more robust interfaces and GUI assimilation."
Still, their efforts and the increasing attention on the category have made BI tools better suited for mainstream business professionals, not statistical black belts.
"The tools have evolved, the platforms have evolved and the ability to manage large amounts of data has evolved -- it's not so off-putting anymore," said William McKnight, president of McKnight Consulting Group, which specializes in data warehousing and business intelligence.
Yet just because the market is opening up doesn't mean there won't be challenges as predictive analytics spreads beyond its core audience. For one thing, there is some work to be done in preparing staff. The team responsible for the advanced analytics must have a keen understanding of the processes and data involved as well as having an analytical/experimental bent. Most companies will not, however, have to hire formal statisticians to handle their needs but rather can get by with traditional business analysts or with help from BI consultants specifically trained in this domain.
As with other BI initiatives, data quality also remains a huge hurdle.
"Just like with BI, if the data is bogus then the predictions are bogus," said James Kobielus, senior analyst at Forrester Research Inc., who estimates that 75% of the work involved in any of these projects is spent doing data preparation and cleansing.
In addition, companies that have jumped in with predictive analytics often do so in a stove-piped fashion, where individual departments or divisions create their own models even if they're accessing data from an enterprise data warehouse source.
"This is clearly a waste of resources and bandwidth, and companies aren't sharing best practices," Kobielus explained. "What's happening now to some degree is a push towards convergence of data mining operations around an enterprise data warehouse."
With this scenario, which he admits is a new frontier even among data warehousing leaders, the data analytics, regression and scoring models and data mining logic are moved to the enterprise data warehouse, where they can be leveraged throughout the organization.
"This lets organizations share a common set of tools and best practices while improving productivity and consistency of the modeling team," Kobielus explained.