Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

How to evaluate and select the right BI tools

Some advice for organizations trying to settle on BI tools criteria: Figure out which product features are must-have and which are just nice-to-have.

Selecting the business intelligence analytics tool that's the best fit for the enterprise is critical to the success...

of any BI project. This process includes gathering and prioritizing business intelligence requirements, as well as determining use cases and tool categories and styles.

Let's define the key features and functions used to evaluate and select BI tools. These criteria can be utilized when it's time to create a request for proposal (RFP).

BI tools selection steps and evaluation criteria

Although industry analyst product reviews can be a good source of introductory research, particularly for those not familiar with the overall market, these reviews are often oriented toward selecting the product with the most features. Organizations should instead prioritize requirements and select the business intelligence tool that's the best fit for its use cases -- more on that below -- that meets its budget, and that can be implemented given its resources and skills.

To simplify the process, classify the features and functions as must-haves, nice-to-haves and will not use.

  • Must-haves.This classification should be unambiguous. In other words, if the product doesn't have this particular feature and meet specific requirements, it should be eliminated from further consideration.
  • Nice-to-haves.Although nice-to-have features aren't required, they're often the differentiators when selecting a product.
  • Will not use.Many business intelligence products have a laundry list of features that a company may never use. In that case, don't waste time examining those aspects of the products during the evaluation process.

One caveat: Although a product may have the features that meet certain criteria, there may be special considerations for how you obtain those features.

For example, in order for BI tools to provide these features, are any of the following required?

  1. custom coding;
  2. the purchase of an add-on product from a third party; or
  3. a specific product edition, such as an enterprise versus a basic edition.

These conditions all mean additional time and expense. To ensure an objective evaluation and avoid surprises if you select this product, determine how to identify and factor the additional time and cost into product comparisons.

Overall BI features: The must-haves

The following are often must-have features for organizations:

Data sources. Access to various databases and file types, such as comma-separated values files, text, Excel and XML, are basic staples of all BI tools. Increasingly, BI tools are providing access to specific applications, such as Salesforce or NoSQL databases. Specific needs will determine if these features are must-haves.

Data filters and drilldown. The product should enable you to filter the contents in a tabular report or visualization by data values. Filtering is provided by features such as drop-down menus, search filters and slicers. The product should also enable the user to drill down from summarized to more detailed data and then drill up -- i.e., go back to where they started. This is essential in both tabular reports and visualizations.

Web-based client user interface. The product's client user interface for the BI consumer role should be web-based. This has become an industry best practice, as it's more cost- and resource-effective for administration, support and deployment than desktop-based interfaces. It's a nice-to-have feature if the BI application creator and administrator roles are also web-based versus a desktop-based client application.

Independent and interconnected mashups. When the business intelligence style enables a single-screen display of multiple visualizations, including tabular reports, the software should allow these visualizations to be either independent of each other or interconnected. If they're interconnected, data filters and selections will affect all the visualizations; for example, if a particular attribute is selected, all the visualizations will share that attribute.

Visualizations. BI tools must provide bar, line, pie, area and radar chart types, as well as the ability to mix and match various combinations.

Security. All business intelligence products require both user and user role-based security, designating who can create, modify, publish, use and administer the BI applications. Some may require the BI product to integrate with the operating system or other pre-existing security applications. BI security often involves using the product's security along with a combination of mechanisms from the operating system, networks, databases and the source system.

Microsoft Office data exchange. The product must be able to import and export data with Microsoft Office products, especially Microsoft Excel.

Print and export. The product must enable you to export print visualizations and tabular reports to PDF or other graphics. Tabular reports need to be exportable to text files at a minimum and, preferably, to spreadsheets.

Must-have features specific to self-service BI uses cases

There are several must-have features that are specific to self-service BI use cases. These are unique because they provide more data management functionality for the business person creating an analytical application than for an information consumer who is relying on prebuilt business intelligence applications with prebuilt integrated data. These features include:

Select data for analysis. BI tools must enable the user to select the data used in decision-making analysis and present it as a pivot table-style interface where dimension attributes are placed in rows and columns, measures are selected, and filters are applied.

Data blending. The product must permit the user to blend data from various data sources. This includes accessing the data and mapping or creating relationships with data from multiple sources.

Create measures. The product must enable the user to create and save measures or calculations for use in analysis. These are also referred to as performance measures or key performance indicators.

Create hierarchies. The product must enable the user to create dimensional hierarchies, such as by geography or product, to group and summarize data. This establishes the drill-down paths.

Save queries and analysis. The product should enable the BI user to save the data filters, selections and drill-down paths used in decision-making analysis so they can be reused.

Overall BI features: The nice-to-haves

These features are often the differentiators when selecting BI software.

Create and publish by business users. The product must enable the user to save and share his or her analysis with other BI consumers.

Context-based filters. Filters will list only the choices that have values that fit the current selection of facts and dimensions.

Context-based visualizations. Only visualizations or chart types that are relevant to the data selected will be listed as options.

Advanced visualizations. More advanced visualizations include heat maps, scatter plots, bubble charts, histograms, geospatial mapping and combinations of each of these, such as bubbles on a map. The best mapping capabilities will leverage city, state and country attributes in the data rather than force the inclusion of longitude and latitude.

Collaboration and social interaction. BI tools enable the creation of a business community that can share and discuss their decision-making analysis. This would include annotating analysis to share observations and social media, which can enable discussion threads or chats.

Storyboarding. Business analysis often involves a process or workflow to analyze different data from different perspectives. Storyboarding enables a series of reports or visualizations to be tied together in a workflow that can be shared.

Microsoft Office real-time data integration. Beyond simple import and export, the product should provide real-time data integration with Microsoft Office products, which enables business people to embed analytics from the business intelligence tool into a PowerPoint or Excel presentation, for example, and refresh it automatically as the data is updated.

Mobile version. BI tools should be able to differentiate between viewing BI applications on a web browser on a mobile device versus a mobile BI application.

In-memory analytics. The product should pull data into an in-memory or locally cached data store. In-memory columnar is an increasingly popular feature that enables very fast analytics once the data is loaded.

Offline updates. BI tools, when storing copies of the source data in an online analytical processing (OLAP) cube or in-memory columnar data store, should enable business users to schedule automatic data updates.

Performance monitoring. BI applications that monitor report and data usage enable a BI group to improve analytical performance for the business, eliminating bottlenecks and enabling users to assess infrastructure needs.

Business intelligence platform administration. Although all BI software should provide code and version management, there are many application development features, such as team development and user administration, that are useful for larger BI deployments.

Other considerations

Establishing the scope of the BI project in terms of how many people will use it and what data they will need to access is the foundation of creating the selection steps and criteria.

Monetary considerations, such as the anticipated budgets for the initial project, sustaining the BI program and expansion the following year, are also key factors in the selection steps and criteria.

Establishing the scope of the BI project in terms of how many people will use it and what data they will need to access is the foundation of creating the selection steps and criteria.

Although one wouldn't go to an architect and just ask to design a house without talking about the size, type of rooms and budget, too many BI evaluation projects have started without any scoping or budgetary boundaries, resulting in time wasted examining BI applications that don't fit a company's need or budget.

The following are often included in evaluation criteria, but since they're very subjective, it's important to provide clear definitions of what the comparison will be based on.

  • Ease of analytical use.There should be different criteria defined for each type of user, such as information consumer, business analyst and IT.
  • Ease of creating BI applications.There should be different criteria for each type of analytics creator, such as business analysts and IT.
  • Speed of access.Query performance will vary based on the complexity of the queries and the amount of data involved. Dashboards with multiple visualizations will need to get query results from many queries. The best practice is to create several prebuilt query scenarios and compare how each product performs based on these specific examples. The worse practice is to just arbitrarily rate the speed.
  • The best practice is to establish a testing environment to determine scalability in terms of both the number of concurrent users and data metrics, such as volumes, variety and veracity.
  • Does the company prefer on premises versus cloud, open source versus commercial software, operating systems, or other infrastructure options?
  • There should be separate criteria for BI user versus administration training. Training may include in-person classes, online classes -- live or prerecorded -- or web recordings for specific features or processes.
  • There should be separate criteria for BI user online help versus technical documentation.

Selecting the right BI category and style

The following BI use cases can assist you in selecting the appropriate business intelligence category and style. Many enterprises will have multiple BI use cases, so it's important in those situations to match the right BI category and style to the right business users.

Although it may seem that giving every style to every business person is a good thing, the reality is that it will likely overwhelm them and prevent them from using the BI tools effectively -- or at all. It's like the story of Goldilocks: The software shouldn't be too hot or too cold, but just right.

BI use case: Operational snapshots. The business needs a recurring snapshot of operational performance on a daily, weekly, monthly or quarterly basis. The performance measures and the data that need to be reviewed are well-defined, and the analysis work typically involves period-over-period performance comparisons or trends.

Business people may filter data based on agreed-upon criteria, but they primarily want to do some analysis quickly and then get back to their jobs. To make that feasible, they need tabular reports and easy-to-grasp graphics, such as basic bar and sparkline charts. Data consistency is key, with IT integrating data as necessary in the background.

Recommended business intelligence category and style: Guided analysis/reporting tools.

BI use case: Limited exploration. Similar to the first use case, the business needs a consistent set of data and performance measures available on a recurring basis, but in this scenario, the users want to do limited data exploration themselves. They still primarily require a mix of basic business graphics and tabular data, but they also need to be able to drill down into the information for further analysis.

That combination can best be accomplished through BI dashboards that are a mashup of several related graphics with underlying detailed data that business users can access, filter and analyze. Dashboards have traditionally been created by IT, but an increasing number of them are now being built by business analysts using data discovery tools.

Recommended business intelligence category and style: Guided analysis/dashboards.

BI use case: Packaged applications. Corporate performance management (CPM) applications are built to support either specific industries, such as healthcare, or business functions, such as finance. The most prevalent applications built into CPM are forecasting, planning and budgeting. CPM applications are often linked to specific enterprise operations systems.

Although this is a niche market, if the application matches the enterprise's needs, then there are significant advantages to buying packaged software that incorporates industry best practices into its data and analytical processes. However, the tradeoff may be a lack of flexibility to match the enterprise's unique needs.

Recommended business intelligence category and style: Guided analysis/corporate performance management.

BI use case: Spreadsheet integration. Many business people use spreadsheets to gather data from various sources, integrate the data and then create reports. The gathering and integration processes are often time-consuming, involve many manual processes and require integration techniques that may not be known to most business people. Spreadsheet integration tools expand beyond simply importing data in the same manner as CSV or text files by accessing integrated data sources and enabling data blending.

In this scenario, business people can continue to use spreadsheets as BI tools, but rely on the spreadsheet integration tools to perform the gathering and integration that was time-consuming and error-prone. Business users often favor this approach because they feel more productive using spreadsheets rather than different BI tools to analyze data. If they're devoted spreadsheet users, it will be difficult to get them to shift -- and, quite frankly, they may be able to do more advanced decision-making analysis with spreadsheets than BI software can offer.

Recommended business intelligence category and style: Guided analysis/spreadsheet integration.

BI use case: The relevant data isn't known prior to analysis. In this use case, not all the data sources or performance measures are predefined, and most analysis is done just once. The users are business analysts who are self-sufficient in regards to getting and analyzing data -- they're both data- and analytics-savvy. In fact, they may have created data shadow systems for their business peers and are the subject matter experts for IT when they're examining data source systems.

Data shadow systems are spreadsheets used to gather, integrate and analyze data from various sources. They often grow from a simple spreadsheet used to gather data from a single data source into a complex application involving hundreds of spreadsheets or worksheets pulling data from a myriad of data sources.

BI ad hoc query tools were built for business analysts who need to do intensive data exploration to determine what data is relevant to their decision-making analysis.

Recommended business intelligence category and style: Self-service BI/ad hoc analysis.

BI use case: Performance measures aren't defined prior to analysis. In this case, there's a known collection of data sources, but you may need to define the performance measures while the analysis is being done. As above, the analysis work may be one-time in nature and typically requires savvy business analysts.

However, these analysts aren't proficient in using query-based tools -- they're more comfortable with spreadsheets. The best match for them is OLAP or pivot table analysis tools that are just like working in a spreadsheet.

Recommended business intelligence category and style: Self-service BI/OLAP or pivot table analysis.

BI use case: Not all relevant data sources or performance measures are known. This is another example in which the user doesn't know all the relevant data sources or performance measures when starting with analytics applications, so business analysts will need to blend in data and define performance measures during the analysis process. In doing so, they may need to engage in extensive data exploration work -- but they typically aren't inclined toward using either an ad hoc query tool or OLAP-oriented software.

They prefer an easy-to-use BI product rather than one with a steep learning curve, which points toward data discovery tools. Such tools may include data visualization capabilities, but usually include built-in dashboard capabilities that business analysts can use to deliver analysis results to business managers on either a one-time or recurring basis.

Recommended business intelligence category and style: Self-service BI/data discovery.

With the evaluation criteria, requirements and use cases in mind, it's time to select a shortlist of product candidates and proceed with the RFP process.

Dig Deeper on Business intelligence software

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What are some of your must-have features for a BI analytics tool?
What are your criteria for BI tools and what are your must-have features?
Great article! Another effective way to evaluate and select the right BI analytics tool for your company is by reading user feedback. Your readers may find real user reviews for all the major BI tools on IT Central Station to be helpful: https://goo.gl/OHnmHx

Users interested in BI tools also read reviews for Oracle OBIEE. This Business Analysis Manager writes in his review, “I can make a comparison with Business Objects and MicroStrategy. Even though these are excellent tools for business analysis, Oracle OBIEE has allowed the company to approach data analysis in a structured and shared way favoring the growth of analytical expertise across the entire company.” You can read the rest of his review, as well as explore what others have to say about OBIEE, here: https://goo.gl/xA91Ps