This article originally appeared on the BeyeNETWORK.
In early 1990s, we worked with a large telephone company, an early adopter of business intelligence (BI) and data warehousing. Even today, the company is an acknowledged business intelligence best practice, and the data warehouse and BI infrastructure we established there still thrives. At the time, the communications industry was still feeling the sting of deregulation, which was driving new competitors into markets that had hitherto been competition-free. This new competitive climate was redirecting the attention of local service companies away from their products and networks and toward their customers.
The newly competitive climate in communications resulted in a frenzy of new customer care and customer acquisition initiatives. Executives at our phone company client were getting restless. They began to realize that their customer data – including purchase behaviors, call detail records, and usage history – could yield a wealth of insight into new product development and service improvements, and that deploying analytics could be a differentiator for sales and marketing in a cutthroat commodity business.
Meanwhile the BI vendors were bombarding the company with pitches and demos. IT was overwhelmed with functions and features, and the business users were still poring over the so-called “green bar report,” a compendium of revenue numbers that, because of the time required to produce it, was obsolete by the time it hit managers’ desks. Inasmuch as the company had a vision for what it could do with information, there was little consensus on where to begin and how to evolve.
The BI Mantra: Start Small, Think Big
We noticed this conundrum at several clients and developed what’s become affectionately known as The Baseline Pyramid. The pyramid represented a de facto evolution of BI which, it turned out, was an apt one for companies across vertical industries. It was deceptively simple, but spot-on in terms of an accurate evolution for BI and the data that enabled it.
When we introduced the pyramid, some BI vendors asked for permission to adapt it to cover the breadth of their tool suites. Even today, you’ll find variations on its evolution (usually some variation of “What happened?” “What’s happening now?” “What will happen?”). At the time, the premise of the pyramid was “start small, think big.” In other words, beginning with a realistic set of value-added business capabilities could not only start BI out on the right foot, but establish a basic data infrastructure that could be built upon over time as reporting and analysis capabilities matured.
True BI best practice companies start modestly, laying a structured foundation for maturity that delivers a series of “quick wins” while making it easy for users to ask for more when they’re ready. A simple, structured path to increasing sophistication and functionality is key. The pyramid’s categories built on one another, leveraging a common data foundation that matured as the BI capabilities themselves evolved. This usually started with a set of standard reports.
Standard reports allow business users to get information on-demand to their desktops. It’s not only a reliable first step in deploying BI, but a good way to prioritize the data necessary to load onto the warehouse. The irony of standard reports was that in the early days of data warehousing, many companies wanted to bypass simple end-user query and reporting and go straight to advanced analytics. They would soon find out that the lack of complete, quality data, not to mention the relative inexperience of their business users, meant that reports really were a better entrée into BI. After all, BI is like most business initiatives. You have to walk before you run.
The BI early adopter companies found was that there was nothing more strategic than putting detailed and meaningful information about customers, products, and sales in the hands of the businesspeople who needed it. And, once the businesspeople saw the results, they’d want even more.
Many users of standard reports slowly realized the possibilities of their newfound data access. Multidimensional analysis – usually deployed using online analytical processing (OLAP) tools – gave more seasoned users the ability to “drill down” on report results, requesting additional details. Thus the sales manager perusing a quarterly report of new business customers could call up additional information on new business customers within each sales region. Multidimensional analysis encouraged businesspeople to expand on the findings (the “whats”) and research root causes (the “whys”). It turned out to be a boon to the average businessperson looking to plumb the well of corporate information for business improvement.
Modeling and Segmentation
As the user community for BI grows, user sophistication grows apace. Most business users will be content with reliable reports and drill down capabilities. As the data gets broader and deeper, opportunities for more advanced analytics emerge. In some cases, this evolution corresponds to the business’ realization that more highly evolved analytical capabilities can deliver even more strategic knowledge – and drive differentiated action. Though they often fall under the general rubric of “data mining,” the subcategories require different processing algorithms.
Predictive modeling isn’t a new concept, having been around as long as the field of statistics. A model is a collection of patterns for a given characteristic, and as such, is represented either through a set of rules or notations, or via “visualization” features. Though not new, predictive models are only as valid and useful as the data that populates them. Companies that develop and enrich the data used for standard queries and multidimensional analysis thereby simultaneously enhance all three capabilities.
Segmentation has likewise been around as a concept. Segmentation divides customers or other data subject areas into certain bands or groupings, in which common characteristics can reveal behaviors. Customer segments can then be analyzed as dimensions using multidimensional analysis, providing less technical users examine the models and segments without having to build the models themselves. Not surprisingly, companies like Royal Bank of Canada that have excelled at segmenting their customers are also acknowledged leaders in customer acquisition and retention.
The pyramid puts modeling and segmentation in the same category because although they involve different underlying techniques, they fall under the rubric of advanced analytics and require specialized skill sets. In most cases they leverage specialized software.
Predictive models represent the basis for the field of predictive analytics, which The Data Warehousing Institute’s Wayne Eckerson defines as “…a set of business intelligence (BI) technologies that uncovers relationships and patterns within large volumes of data that can be used to predict behaviors and events” (Eckerson, Wayne, “Predictive Analytics,” from What Works in Data Integration, Volume 23, The Data Warehousing Institute, 2007). Eckerson calls out the data volumes for good reason: In order to be accurate and valuable, predictive models should access enough detailed data to represent past behaviors in order to indicate future patterns. As the pyramid illustrates, data maturity – including data volumes, data quality, and degree of integration – evolves alongside BI capabilities.
With standard reports, multidimensional analysis, and even modeling and segmentation, the business analyst or knowledge worker has a suspicion of the results she might get back. Unlike the other BI categories in the pyramid, instead of being retrospective (“What happened?”), knowledge discovery is prospective (“What will happen?” “What should we do?”). Knowledge discovery (aka, undirected knowledge discovery) finds hidden patterns in the data. Since we don’t know what questions to ask, we can’t possibly anticipate the answers. These patterns are too specific and seemingly arbitrary to specify, and the analyst would be on a perpetual fishing expedition trying to figure out all the possible patterns and relationships in the database in order to query them.
For instance, a pharmaceutical firm we’ve worked with uses knowledge discovery to examine patterns when certain drugs are taken together, identifying unknown product affinities and behaviors. The company also analyzes sequential product purchases in order to determine whether certain medications are triggering the need for others.
A knowledge discovery activity at a retailer revealed that, of customers who purchased potato chips, 63 percent also buy candy. This so-called product affinity is even more interesting after the discovery that 74 percent of customers who bought both potato chips and candy also bought red wine. Such multi-item affinities could never have been found with a simple query or multidimensional analysis. But armed with this information, a retailer can make a variety of differentiating decisions from sending coupons to targeted snack buyers to placing items more optimally on store shelves. With knowledge discovery, the data never lies. It’s only the interpretation of the data that’s in question.
Analysis Maturity Drives Data Maturity
That the best BI deployments have either consciously or unconsciously evolved their capabilities according to the pyramid’s structure is uncanny. The more information users get, the more they want, and the more of it they use, the more sophisticated they become. This phenomenon, combined with improvements in dashboard and visualization tools, has caused companies to turn the mirror back on their data.
As the left side of the pyramid shows, data maturity enables BI sophistication. After all, the results of data mining are only as valid – and as useful – as the data that informs them. Usually, the inverse is true as well.
But large data volumes aren’t the only determinant of data maturity. Well-meaning companies and vendors alike have made the mistake of confusing data integration with data consolidation. Simply replicating detailed data from the source systems onto the common disk drives of a data warehouse won’t enable BI as fully as data that’s modeled and integrated. The increase in adoption rates of master data management solutions bears this out, as many companies pursue data integration at the operational level in order to ease the burden of, among other things, complex extract, transform and load (ETL) programming and inaccurate, latent data. (According to IDC, the master data management market will grow to $10.4 billion by 2009, representing a compound annual growth rate of 13.8 percent.) The business case for master data management (MDM) has as much to do with improving the quality and trustworthiness of corporate data as it does with multisystem integration.
Nevertheless, BI has emerged from the back office to the desktop, and businesspeople at most companies have only vague memories of what they did without it. In Part 2 of this article, we’ll focus on who those business users are and what they’re doing with their hard-won information.
Jill Dyché is a partner co-founder of Baseline Consulting, a technology and management consulting firm specializing in data integration and business analytics. Jill is the author of three acclaimed business books, the latest of which is Customer Data Integration: Reaching a Single Version of the Truth, co-authored with Evan Levy. Her blog, Inside the Biz, focuses on the business value of IT.