“Big data” is arguably the big buzzword of 2011. The term, as described by the Stamford, Conn.-based research firm Gartner Inc., refers to data growing exponentially in volume (transactional data), velocity (streams of data) and variety (email, Web server logs, video and image). Throughout the year, vendors touted new and refreshed products and strategies that highlighted ways to store, manage and, perhaps most significantly, analyze big data.
As 2011 winds down to its final days, the big data challenge continues to generate momentum. Gartner, for example, added big data to its 2011 hype cycle and has called it one of the top 10
strategic technologies for 2012, stating, “The size, complexity of formats and speed of delivery exceeds the capabilities of traditional data management technologies; it requires the use of new or exotic technologies simply to manage the volume alone.”
The following is a roundup of stories produced by SearchBusinessAnalytics.com in 2011 that drill down into the big data phenomenon -- from job titles and anticipations to evolving tools and new technology.
5. Tapping into transactional data and more
What is big data? If you’re still not clear on it, start with this video series from the 10th annual Pacific Northwest BI Summit.
William McKnight, president of McKnight Consulting Group, begins by saying: “It’s a class of data that we haven’t historically managed very well. It’s this unstructured world, it’s Web logs, and it’s this data we’ve left behind in our enterprise management system.”
The McKnight interview is one of four short videos that feature industry experts discussing big data and detailing the challenges businesses face when getting started.
4. Advanced data visualization is key
According to recent research by The Data Warehousing Institute, advanced data visualization is viewed by business users, IT professionals and consultants as the No.1 tool when analyzing big data. It allows complex, multi-varied data to be explored for patterns, articulated clearly and used quickly. But when data sets edge businesses into the “big” territory, visualizations can become bogged down or overly simplistic. Finding the right balance between the two is key.
3. Big data and advanced analytics
Several of this year’s most popular big data stories also featured predictive and advanced analytics. Doing advanced data analysis on big data sets can provide opportunities to tease out the more granular details in structured data, but there’s also the promise of utilizing unstructured data from social media sites or user-generated content. But businesses are finding limitations to traditional analytics approaches and traditional relational database management systems; now, early adopters are beginning to create a hybrid analytics environment, which may include massively parallel processing technology like that offered by Hadoop, to take on these multi-varied kinds of data.
2. In-memory analytics appliances
In December, SAS Institute Inc. released its high-performance computing platform, which marries SAS analytics software with hardware from its partners Teradata and EMC Greenplum. The new appliance is equipped to handle big data due, in part, to its use of in-memory analytics -- a growing trend also at the foundation of SAP’s HANA as well as Oracle’s Exalytics.
1. The data scientist and big data
While it has yet to be fully embraced by the business world, vendors, analysts and businesses are increasingly employing the title data scientist. The term is typically invoked to describe a kind of employee with a statistical, computer science or machine learning background capable of -- and interested in -- discovering new patterns or isolating important nuggets within the data and relaying the information back to the enterprise. Data scientist tends to be used synonymously with big data to indicate the kind of expertise needed when analyzing the growing amount of multiple kinds of data produced at a rapid pace. Those skills are hard to come by, and according to a May 2011 McKinsey & Co. study, will continue to be so.