For years IT and business intelligence (BI) teams have focused on repetitive structured transaction data. But structured data represents only about 20% of the overall data in most companies. That means they’re missing out on the business insights that can be found in the other 80%—the unstructured data. It’s time for a change.
New technology allows unstructured data to be included in the decision- making process. For example, standalone text analytics tools can be used to look for patterns in text data and assess its meaning and sentiment. Organizations can also now place text in standard relational databases, so it can be stored in data warehouses for mainstream BI tools to analyze.
Having this previously unstructured data available for analysis presents valuable information that can be used to make better decisions. In addition, new business opportunities can be uncovered that would never see the light of day by analyzing conventional record-based data only.
Here are a few examples of the kinds of text data that companies can now use more effectively:
Business management has always had the ability to compare and analyze a small number of contracts. But without the ability to incorporate text data in BI and analytics processes, there was no way to analyze 10,000 or 100,000 contracts at a time. When a company has the ability to look collectively at the text in corporate contracts, management can answer questions such as these:
- How many contracts expire in six months?
- How much liability do we have?
- How much could we save if we had a single master contract with a customer?
Health care data
Medical data is permeated by text that was difficult to analyze in the past. But now it is possible for hospitals and health care organizations to capture, store and analyze huge amounts of text data in an automated manner.
Many corporations depend on email as an essential part of the business. But typically, once an email is read, it effectively goes into the corporate trash can. That’s a shame because emails often contain important information about business transactions, customer attitudes, complaints, product malfunctions and so forth. Now email messages can be saved, organized, filtered and analyzed.
Warranty claims data
Warranty claims contain a large amount of information in text form. Certainly that information is important to customers who have had problems with products or services, but it’s also extremely valuable to a company’s engineers and product designers. Properly analyzed, warranty claims data can help them detect—and then correct—product flaws and weaknesses.
The U.S. economy was thrown into a recession in late 2007 partly because of risky mortgage-lending practices by banks. In addition, the number of loan applications that were being generated before the economic crash overwhelmed the capacity of bank workers to analyze them manually. Now, by using BI tools to analyze text from loan applications, banks can more easily assess the underlying value and risks of their loan portfolios.
Call center records
In many organizations, call centers are at the heart of interactions between consumers and the company, and they generate large amounts of text data. In addition to the customer service representatives who deal with customers, there are many other parties in a company who can benefit from the ability to analyze that information. For example, senior management can get feedback on products and services and how they could be improved, or they can gauge the interest in new products and strategic directions. Being able to analyze written records of call center conversations opens a new avenue of insight for companies.
System, network and Web server logs hold cryptic but useful text-based information. For example, suppose a significant systems event occurs at an organization. IT managers can use the log data to examine and analyze the activity that preceded the event and determine whether there were any warning signs or predictors that can guide future actions. In doing so, companies can become proactive rather than reactive. And the list goes on. Practically everywhere you look in any company, you find text. And everywhere you find text, there is a latent business problem waiting to be solved through BI and analytics processes.
About the author:
W. H. “Bill” Inmon, known as “the father of data warehousing,” has published more than 40 books and 1,000 articles on data warehousing and data management. Inmon speaks frequently at seminars and industry events, does consulting work and is president and chief technology officer at Forest Rim Technology LLC, which develops software for integrating text data into data warehouses. Email him at email@example.com.
- Business Intelligence Software: Which one is the best? –IT Central Station