Essential Guide

Browse Sections


This content is part of the Essential Guide: Guide to big data analytics tools, trends and best practices
Evaluate Weigh the pros and cons of technologies, products and projects you are considering.

Stream processing not such a big-data deal for some users

Many data streaming applications don't involve huge amounts of information. A case in point: an analytics initiative aimed at speeding the diagnosis of problems with Wi-Fi networking devices.

Stream processing vendors tout their ability to handle millions of transactions or records per second -- but applications with a data load of that magnitude are a niche within a niche. Typically, much less information is involved, even in big data applications. "Streaming can be pretty relative," said Nick Heudecker, an analyst at consulting and market research company Gartner Inc. "In reality, probably most applications are in the tens of thousands [of transactions] per second."

Meru Networks is an example of a company that's deploying stream processing technology to help it analyze a relatively small amount of data. The technical support team at Meru, a maker of Wi-Fi networking devices in Sunnyvale, Calif., has been using analytics tools from software vendor Glassbeam Inc. since late 2013 to check log files from network controllers at customer sites in order to help diagnose and resolve problems. The machine data was being processed in batch mode -- but now Meru is adding more real-time analytics capabilities for some of its customers through a pairing of the Glassbeam software and the Apache Spark processing engine's data streaming module.

Joe Limprecht, Meru's technical support manager for the Americas region, said automated scripts running in Spark Streaming will be able to pull data every five to 10 minutes from controllers at Tier 1 support customers experiencing technical problems that have been escalated for higher-level investigations.

The output from the log files isn't huge: Each contains about 60 MB of data, and customers generally have 20 to 30 controllers installed, Limprecht said. But the goal, he added, is to enable Meru support engineers who are trying to find fixes for thorny issues "to essentially see live data when we're talking to customers." For example, they might be able to analyze the streaming data to pinpoint traffic spikes that require the addition of more wireless access points to a customer's network.

Craig Stedman is executive editor of SearchBusinessAnalytics. Email him at, and follow us on Twitter: @BizAnalyticsTT.

Next Steps

Streaming data can help bring big data into real time

Medical company turns to streaming data to spark real-time analytics

Dig Deeper on Big data analytics

Join the conversation


Send me notifications when other members comment.

Please create a username to comment.

What's your best advice for deciding if stream processing technology is right for an organization?
Great article -- thanks for the content. 

You might consider that the value of stream (or event) processing extends well beyond the capability of capturing a single raw stream of data from a source system.  In fact the strength of a stream processor is...

* The ability to apply logic (and possibly advanced analytics) to the stream in order to selectively retrieve the data that is necessary (thereby reducing post processing)

* Identifying business or process events that are only detectable when reviewing indicators from multiple sources

* combining and integrating data from multiple disparate sources to provide a more complete picture of a particular business or process situation.

Complex event and complex stream processing shouldn't be considered as a replacement for other data extract or data capture technologies; their strength is to enable analysis and processing of transactional content to simplify data usage by the data consumer