News Stay informed about the latest enterprise technology news and product updates.

Text analytics, search bolster business intelligence software stack, says TDWI

Text analytics, search and other unstructured data technologies can bolster business intelligence software, says TDWI.

Two unstructured data technologies can seriously bolster business intelligence software stacks -- and one can be a "no-brainer" to implement, according to an analyst.

For more on business intelligence, unstructured data and search
Learn how business intelligence vendors are addressing search

Read five myths about business intelligence and search
Text analytics and search have both been around for a while, but both were recently examined in the context of BI in a study and survey of 370 data management professionals by The Data Warehousing Institute (TDWI), based in Renton, Wash. The two technologies are different but related because they're typically applied to unstructured data, such as that found in documents, emails and text fields, explained Philip Russom, senior manager of research and services with TDWI and author of BI Search and Text Analytics: New Additions to the BI Technology Stack, published in the second quarter of this year. Companies are increasingly integrating these technologies with BI software, he said, albeit in very different parts of the stack.

"Text analytics is a very deep technology piece that end users never see, whereas BI search is something every report consumer can see and use," Russom said.

Business intelligence search increases productivity and self-service

On the front end, BI search enables productivity and efficiency gains, by helping end users find reports and information faster, Russom said. Some companies seem to agree, with 62% of TDWI's survey respondents rating the perceived business value of BI search as high or very high. Though only 9% of respondents had deployed BI search technology, 15% are in the design or implementation phase, and 40% are in the exploration phase. Russom is "amazed" that more companies haven't implemented BI search.

"Search capabilities are now built into most BI platforms. [Many] have switches -- you just turn on the switch and it enables search capabilities for their BI platform," Russom said. "If it's that easy to implement, just go ahead and do it. In fact, with many licenses, you don't actually pay extra money for it."

Companies that haven't implemented BI search may be strapped for time or may be dealing with competing priorities, he said. But he thinks organizations might move BI search to the top of the queue if they considered its potential long-term benefits, such as reducing end user requests for new reports or tech support.

"BI search enables a wide variety of self-service capabilities that offload work from IT to the report-consumers," Russom said.

Text analytics increases knowledge and insight

Organizations can gain valuable insight by extracting and analyzing information from unstructured data, Russom said. In the BI context, text analytics is on the back end, part of the underlying data integration infrastructure, he explained. It can be complex to implement, in part because of the development work required and the volumes of unstructured data at most companies -- though the volumes may not be as big as previously thought, he found.

An oft-cited statistic is that, on average, unstructured information makes up 80% of the data in an enterprise, Russom said, but this TDWI survey indicated that 31% of enterprise data is unstructured, 22% is semi-structured (as in XML, etc.) and 47% is structured. Though it's a change from previous figures, it doesn't change the importance of unstructured data to BI, he said. A company's view of corporate performance is incomplete without the contextual information that's provided by unstructured and semi-structured data.

One example of this was an insurance company Russom surveyed that uses text analytics to examine call center records. Call center representatives entered a status code (structured) for each call, which helped the company measure metrics such as customer satisfaction. To gain insight beyond the codes, the organization used text analytics to look at the comments field (unstructured text). The company found that customer satisfaction metrics looked very different depending on whether they considered the status code alone or combined that information with data extracted from the text comments field.

"In the text, you could find things about the customer that the structured data from that call did not capture. So they were actually able to create some new metrics that had to do with softer quantification of customer satisfaction," Russom said.

Some companies seem to recognize the importance of using text analytics in BI, the survey found. It said that 51% of respondents rated the business value of text analytics as high or very high. As for usage, 6% reported having text analytics already deployed, while 7% are in the design or implementation phase, and 43% are in the exploration phase. Russom was not that surprised by the low implementation numbers.

"Text analytics is a full-blown development project. It is not the kind of thing that you can install and be using in a week, like BI search. Text analytics takes a lot of design time before you even select a tool. And, because of the development work, it involves a lot of payroll costs," he said.

The costs and complexities of text analytics can be worth it, he said. Early adopters have used it for warranty analysis, fraud detection and call center analysis, but many companies can benefit.

"If you don't have data in your warehouse representing information that originates from unstructured sources, your data warehouse may be a single version of the truth," Russom said. "But it's not a complete version of the truth."

Dig Deeper on Business intelligence software

Start the conversation

Send me notifications when other members comment.

Please create a username to comment.