BI architect has new options to meet growing data, analytics demands

Designing a business intelligence architecture that can meet the expanding needs of BI users is an exercise in finding the right combination of tools.

At one time, business intelligence architects faced a relatively simple choice between two flavors of online analytical...

processing technology: multidimensional or relational. Now the BI flavors available to organizations are much more plentiful, and the architectural building blocks of BI platforms and underlying data warehouse systems have multiplied.

The diversity of options sometimes can be a drawback if it results in higher technology costs and increased management complexity. But it's primarily a welcome situation for the BI architect as well as BI managers at a time when data volumes are growing rapidly and the variety of data types being collected for analysis has exploded in many companies.

BI architecture is not about imbibing software.

Krish Krishnan,
analyst, CBIG Consulting

In addition, end-user requirements for BI data and reports have expanded and become more complex as BI and analytics have taken on an increasingly central role in business planning and operations. And, in many cases, business users are looking for mobile BI capabilities or self-service tools that let them bypass canned report templates and IT-developed queries in order to slice and dice data according to their own needs and interests.

"The usage of analytics has skyrocketed, and self-service is a huge trend," said Boris Evelson, an analyst at Forrester Research Inc. in Cambridge, Mass. He added that recent technology advances supporting more freestyle data analysis represent a "night and day difference" from BI processes as they were done in the era when users worked exclusively from fixed schemas created by IT.

As a result, BI architects looking to meet new user demands and requirements, while also having to support larger and more variable data sets, often find that established reporting, dashboard and BI tools need help, according to Evelson.

In the case of big data environments, for example, large amounts of structured or unstructured data won't fit into spreadsheets, and it takes too long to run SQL queries against all the information, Evelson said. The required help can take the form of in-memory analytics software and other data exploration and discovery tools that provide search-style capabilities and don't require users to work from predefined data models, he said. Hadoop clusters and associated tools, such as the MapReduce application programming framework and NoSQL databases, can also be called on to aid the big data analytics cause.

Things for a BI architect to think about

There's an abundance of other choices available to the BI architect. In November 2012, Evelson and Forrester colleagues detailed a BI reference architecture with more than 35 components across six separate layers, ranging from data sources to data-delivery mechanisms. The reference architecture also includes seven supporting elements that span all of the various layers, including big data, integrated metadata and information lifecycle management.

Evelson and Forrester analyst Noel Yuhanna acknowledged in a report they co-authored about the reference architecture that it "may not look pretty." But neither do most real ones: "BI architecture is never simple, and this is especially true in large, heterogeneous enterprises with global reach," they wrote. "Such enterprises always have more than one enterprise data warehouse, hundreds of data marts and several BI platforms."

Even the back-end technologies that feed data to BI platforms have significantly expanded beyond the relational data warehouses and data marts that have long been familiar to architects. Besides Hadoop file systems and NoSQL databases, new infrastructure elements can include columnar databases, data warehouse appliances and in-memory database systems.

"At one time, the options for analyzing data were limited to the products of a few big players and a handful of best-of-breed startups. Now there are a lot of options out there," said Joe Caserta, president of Caserta Concepts LLC, a New-York-based data warehouse consulting and training company. Caserta is also co-author -- with BI and data warehousing consultant Ralph Kimball -- of The Data Warehouse ETL Toolkit.

Evelson cautions, though, that the surfeit of available pieces shouldn't invite overindulging. If you field too many architectural components, you easily could end up with shelfware -- i.e., unused technology. "Large companies often have more tools than they really need," he said. "Smaller businesses need to understand that if they can get 80% of what they think they need, that's better than 500%."

Architectural overload weighs down BI systems

Krish Krishnan, an analyst at CBIG Consulting in Rosemont, Ill., agrees that more can be less when it comes to business intelligence architecture development.

More on BI and data warehousing architecture

Check out some key business Intelligence case studies

What counts? What you do with the data

Read about the logical next step for the data warehouse

Diverse architectures are required in most cases, Krishnan said -- particularly nowadays, with data warehouse environments being extended to include pools of big data and the idea of a homogeneous single enterprise BI platform no longer valid in many organizations. "There's more coming out of the woodwork every day, and heterogeneity is guaranteed," he said.

But, like Evelson, Krishnan warned that designing a successful BI architecture "is not about imbibing" software. Being able to inventively combine architectural parts is the key to success, he said, noting that BI teams must also be prepared to adapt architectures to fix data problems and meet new business requirements.

"You cannot commoditize [architecture design]. Each organization is different," Krishnan said. As a result, BI architects "need to think outside the box," something he doesn't see happening enough in companies.

Evelson and Yuhanna similarly said in their BI reference architecture report that the fluid nature of business -- and data -- mandates that architectures be flexible enough to handle necessary deviations from the standard design. "The ability to deal with exceptions rather than fighting them is often what separates a successful BI environment from a failed one," they wrote.

Follow SearchDataManagement.com on Twitter: @sDataManagement

Dig Deeper on Business intelligence architecture and integration