As if the term big data wasn’t confusing enough, now analysts are trying to size the market and calculate the compound annual growth rate (CAGR) three and even five years out. But the forecasts can differ dramatically depending on who does the analysis.
Wikibon, an open source research and advisory organization based in Marlborough, Mass., estimates the big data market will grow from $5 billion this year to $53.4 billion in 2017. By 2015, it gauges the market will have crossed the $30 billion mark.
Compare that with a report by IDC in Framingham, Mass., which projects a more conservative growth rate. Starting at $3.2 billion in 2010, IDC estimates the big data market will hit $16.9 billion in 2015.
Despite the differences, both reports indicate significant market growth over the next few years, which means big data is not going away any time soon.
What is ‘big data?’
“Big data” is used to describe the voluminous amount of structured, unstructured and semi-structured data a company creates -- data that in many cases would take too much time and cost too much money to load into a conventional relational database for analysis.
Read more from the Whatis.com definition of big data.
Defining big data
Projecting the future of any market isn’t an exact science, but big data may be trickier than most.
“Gartner hasn’t really gone there yet,” said Rita Sallam, research director for Gartner Inc. in Stamford, Conn. “We’re still trying to quantify what tools, technologies and services actually make up the big data market. We’re in the process of thinking through what that sizing might be.”
Differences between the IDC and Wikibon reports hinge -- at least partly -- on how analysts defined what they were measuring. While IDC established a set of data parameters and studied deployments, Wikibon pored over vendor revenue figures from big data product sales.
“It was a part of our goal to be very explicit about what we’re counting and what we’re not counting,” said Dan Vesset, vice president of IDC’s business analytics research and the lead author of the March report Worldwide Big Data Technology and Service 2012-2015 Forecast. “People can either agree or disagree with the methodology, but they will know exactly what our view was.”
IDC studied current and expected deployments where the data collected is more than 100 terabytes. That includes large data sets, smaller data growing at more than 60% a year and high-speed streaming data collection. In all cases, deployments had to be on a scale-out infrastructure and include at least two data sources or high-speed data collection such as Web server logs. IDC’s report segmented the market to include services, servers, storage, networking and software.
“Sometimes big data discussions center around newer technologies like Hadoop or NoSQL to the exclusion of relational databases, search technologies or text analysis,” Vesset said. “We’re not excluding any type of technology … If [it] can handle the requirements we set, then it was included.”
Wikibon drew a different line. It defines big data as “data sets whose size and type make them impractical to process and analyze with traditional database technologies and related tools.”
“We had an internal debate about [what makes up the big data market],” said Jeff Kelly, principle research contributor and lead author of the February report Big Data Market Size and Vendor Revenues. “We decided to look at it from a hardware, software and services perspective.”
Wikibon studied 2011 revenue numbers from vendors selling what it deemed to be big data products. Vendors unable or unwilling to provide revenue numbers that matched Wikibon’s big data definition left Kelly and his team to “triangulate many sources of information” for the final numbers. The vendors -- 46 in total -- and revenue figures are included in the report, which is available for free.
“There’s no doubt that big data is one of the hottest buzzwords right now,” Kelly said. “Our research is aimed at IT practitioners and business decision makers to help them try and leverage big data.”
According to Vesset’s numbers, the big data market is growing seven times faster than the information, communication and technology market as a whole. And according to Kelly’s report, the top three vendors answering the call are IBM, Intel and Hewlett-Packard, in that order.
While bigger vendors may be selling more big data products, Kelly said a majority of the innovation comes from niche, or so-called pure-play vendors, independent vendors whose big-data related revenue accounts for more than 50% of their total revenue.
But the terrain -- especially where those niche vendors are concerned -- continues to evolve. Kelly noted Vertica, Aster Data and Greenplum, all of which provided next-generation data warehouse technology, are no longer independent entities.
The storage, warehousing and analytics markets have seen this pattern before, but Vesset said the big vendors are adapting to the new changes quickly -- and he expects they will continue to do so.
“Eventually, they will take over,” he said. “But there’s a very long tail of small vendors, especially on the services side. And some of them will probably be the next big thing.”
IDC’s study calculated a separate CAGR for each of its big data market segment. While the findings project storage will see a more than 60% growth a year, it also indicates businesses will spend the most money on services.
More on ‘big data’ software and projects
Find out more about the federal government’s investment in ‘big data’
Read up on how combining ‘big data’ with predictive analytics requires care
Learn how to shop for a ‘big data’ platform
“Because it’s a new area, professional services are required to help businesses out with skills they may not have today,” Vesset said. “It’s primarily a function of an emerging market, an emerging problem set and challenges that are not familiar to many organizations.”
Those same service providers are also feeling their way through the big data unknown.
“Services firms are having the same problem as enterprises,” Kelly said. “The skills are lacking.”
Kelly and Vesset called their market sizing reports a beginning, one anticipating a still immature industry on the verge of growth: How much growth is still debatable.