The key to a successful real-time business intelligence project is to find the right balance between how fast business users want BI data to be delivered to them and how much it would cost to achieve those results, according to Wayne Eckerson, director of research for TechTarget’s Business Applications and Architecture Media Group. When business executives know the expected cost of building and running a real-time BI system that meets user expectations for data delivery speeds, he says, they “can decide how badly they want [the data] that fast” and the plans can be changed as needed to better align with budget priorities and realities.
In the real world, Eckerson notes, data typically doesn’t move in pure real time. What’s usually meant is near real time, which can range from “a few seconds to a few minutes,” he says. With that kind of variability, “perhaps a better term for real time is right time,” he adds. “Right time puts emphasis on the business need.”
Once the business requirements are finalized, there are numerous ways that organizations can start enabling their data warehouses for real-time BI, Eckerson says. For example, they can go from daily loads of data to micro-batch loads every hour or two, or they can integrate other technology such as messaging buses and change data capture tools to help accelerate the delivery of BI data.
In this video interview recorded at SearchBusinessAnalytics.com’s “Delivering Deeper Insights with Big Data and Real-Time Business Intelligence” seminar, Eckerson spoke further with Editorial Director Hannah Smalltree about real-time BI technology options as well as best practices for managing real-time BI projects.
In the video, viewers will learn about:
- What it really means to provide real-time BI capabilities, according to Eckerson
- The technologies that companies are using as part of their real-time BI and analytics programs, and the two main approaches for delivering real-time data
- Some major misconceptions shrouding real-time BI projects
- Case studies showing how companies like 1-800 Contacts are employing real-time BI technology to meet their business goals
- How businesses can migrate to a real-time BI environment
Read the full transcript from this video below: Eckerson: Start looking at real-time BI as ‘right-time BI’
Hannah: Hello and welcome. I'm Hannah Smalltree. I'm the Editorial Director for searchbusinessanalytics.com and related sites. I'm here today at our seminar on delivering deeper insights with big data and real-time technologies. Now I'm speaking with Wayne Eckerson. He's TechTarget's Director of Research, covering business analytics and related technologies. Thank you so much for being with us here today, Wayne.
Wayne: Thanks for having me.
Hannah: Now what does it mean to deliver real-time business intelligence or analytics?
Wayne: Good question. Real-time is a very misunderstood word. From a technical perspective, what a technologist would say is that real-time is something that happens instantaneously. But in the real world or the business world data actually doesn't normally happen that fast. It doesn't move that fast. So, we usually talk about near real-time, which is a few seconds to a few minutes basically. Perhaps a better term than real-time is right time, and right time puts the emphasis on the business need.
So from a technologist perspective what we really want to do is understand the value of data and the time value of that data to the business. Then deliver that to the business in the timely fashion that they want and that they can afford. Those are sometimes two different things. Often times the business will say, "Oh, I want this data immediately." Then you say, well, it might cost this or that, and then they can decide how badly they want it that fast.
Hannah: What kinds of technologies are people using to deliver these programs, or make their programs more real-time enabled than they were before?
Wayne: Well, there's a fundamental question about how to deliver real-time data. I think in my assessment of it, there are two ways to do it. You can do it inside the data warehouse, or you can do it outside the data warehouse. Now, warehouses were designed initially to be batch-oriented systems. So, load it at night or on the weekend for doing more historical analysis.
However, you can real-time enable your data warehouse. You can integrate them with messaging buses. You can do change data capture. You can increase the speed with which you do these batch uploads. Some people call them micro-batches or mini batches.
A lot of companies have moved their data warehouses to being updated every 15 minutes, so that when you run a query every 15 minutes you get the most up-to-date information. That's one way to do it, real-time enable your data warehouse. That ensures all the data stays in one place, and you have a historical record as well as a real-time record.
The other way to do it is to do it outside the warehouse. Set up a separate environment that's a real-time environment. There are many ways to do this, but one of the more popular methods people call complex event processing or stream-based processing.
Typically you're feeding these systems with large volumes of discrete events, sometimes tens of thousands per second. These systems can basically run queries against the data as it's flowing through the system, and then trigger alerts. It's more of an autonomic system. You set it up with rules, and it's going to react, trigger alerts or work flows.
Hannah: What are some of the biggest misconceptions you've encountered around real-time? Maybe it's this issue of near real-time versus actual real-time, but what are some of the things you're hearing
out there in the market?
Wayne: Well, I think one of the misconceptions is semantics. Some people or some vendors when they say real-time B.I., they're actually talking about how fast a query returns a result set. They're talking more about query speed or query performance. They're not necessarily talking about how long does it take to get the data to a decision maker once that data is generated at the point of origin.
That's the latter definition, is how I think of real-time B.I., or operational intelligence. It's basically reducing decision latency. Making data available to the decision maker at the time they need to make that decision, not after.
Hannah: Can you talk a little bit more about some of the more interesting case studies or use cases that you've encountered around real-time technologies?
Wayne: Well, one case study that I talked about in my book I performed on dashboards was an operational dashboard done by 1-800-CONTACTS. They built this operational dashboard in their call center, where people were taking orders for contact lenses by phone and by email and over the web. They essentially wanted to replace reports that no one was looking at with a dashboard that was updated every 15 minutes with new orders and sales.
That was very successful. It saved the company a lot of money. They also included the bonus points that a call center representative would get based on their performance. That was so successful that the executives in the company wanted their own operational dashboard that was updated every 15 minutes with sales and orders, essentially those two metrics. That was also very successful. It helped the executives keep their fingers on the pulse of the company.
Hannah: Finally, how can people migrate their existing B.I. and analytics programs toward a more real-time environment? What advice do you have for people who are seeking to make their programs a bit more real-time enabled?
Wayne: Well, if you have an existing B.I. environment, an existing data warehouse you can start to real-time enable that warehouse. One of the first steps is to go from daily batch loads to micro-batches or mini batches perhaps every hour or every couple of hours. A lot of companies are forced to do that because their windows for loading the data warehouse are disappearing at night. So, many people are actually doing real-time acquisition even though they may not be doing real-time data delivery, just because those batch windows are closing and the data volumes are increasing.
I think a lot of companies now do update their warehouses or a good portion of the data elements in their warehouses on a nightly basis. That's moving to more intraday as we go forward. I think increasing their load frequency is their first step, playing around with change data capture. Integrating your ETL with messaging buses is another way to start trickle-feeding data into the warehouse.
These are all options for existing data warehousing environments. As I said earlier, you could do this outside of the warehouse too, and plenty of people do for specific types of applications. Like fraud detection is one that's typically been custom built. You're taking transactions as they happen. As they happen, you're applying rules to them, very complex rules, and spitting out ones that look like they might be fraudulent for an individual human to assess and evaluate.
Hannah: Well, thank you so much for talking with us today. I'm here with TechTarget's Director of Research, Wayne Eckerson. He covers business intelligence, analytics, data warehousing and related technologies. Thanks so much for being with us here today, Wayne.
Wayne: Thanks for having me.
Hannah: Thanks to you for joining us today. Remember, you can find more articles, videos, and other resources on searchbusinessanalytics.com. Thank you so much for joining us today. We hope to see you here back soon.