Definition

big data analytics

Contributor(s): Mark Labbe, Lisa Martinek and Craig Stedman
This definition is part of our Essential Guide: An architect's guide: How to use big data

Big data analytics is the often complex process of examining large and varied data sets -- or big data -- to uncover information including hidden patterns, unknown correlations, market trends and customer preferences that can help organizations make informed business decisions.

On a broad scale, data analytics technologies and techniques provide a means to analyze data sets and draw conclusions about them to help organizations make informed business decisions. BI queries answer basic questions about business operations and performance.

Big data analytics is a form of advanced analytics, which involves complex applications with elements such as predictive models, statistical algorithms and what-if analysis powered by high-performance analytics systems.

The importance of big data analytics

Driven by specialized analytics systems and software, as well as high-powered computing systems, big data analytics offers various business benefits, including new revenue opportunities, more effective marketing, better customer service, improved operational efficiency and competitive advantages over rivals.

Big data analytics applications enable big data analysts, data scientists, predictive modelers, statisticians and other analytics professionals to analyze growing volumes of structured transaction data, plus other forms of data that are often left untapped by conventional business intelligence (BI) and analytics programs. That encompasses a mix of semi-structured and unstructured data -- for example, internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile phone records, and machine data captured by sensors connected to the internet of things

Big data analytics is a form of advanced analytics.
Big data analytics is a form of advanced analytics, which has marked differences compared to traditional BI.

Emergence and growth of big data analytics

The term big data was first used to refer to increasing data volumes in the mid-1990s. In 2001, Doug Laney, then an analyst at consultancy Meta Group Inc., expanded the notion of big data to also include increases in the variety of data being generated by organizations and the velocity at which that data was being created and updated. Those three factors -- volume, velocity and variety -- became known as the 3Vs of big data, a concept Gartner popularized after acquiring Meta Group and hiring Laney in 2005.


See the four types of big data analytics and what
each is used for.

Separately, the Hadoop distributed processing framework was launched as an Apache open source project in 2006, planting the seeds for a clustered platform built on top of commodity hardware and geared to run big data applications. By 2011, big data analytics began to take a firm hold in organizations and the public eye, along with Hadoop and various related big data technologies that had sprung up around it.

Initially, as the Hadoop ecosystem took shape and started to mature, big data applications were primarily the province of large internet and e-commerce companies such as Yahoo, Google and Facebook, as well as analytics and marketing services providers. In the ensuing years, though, big data analytics has increasingly been embraced by retailers, financial services firms, insurers, healthcare organizations, manufacturers, energy companies and other enterprises.

Big data analytics technologies and tools

Unstructured and semi-structured data types typically don't fit well in traditional data warehouses that are based on relational databases oriented to structured data sets. Further, data warehouses may not be able to handle the processing demands posed by sets of big data that need to be updated frequently -- or even continually, as in the case of real-time data on stock trading, the online activities of website visitors or the performance of mobile applications.

As a result, many of the organizations that collect, process and analyze big data turn to NoSQL databases, as well as Hadoop and its companion tools, including:

  • YARN: a cluster management technology and one of the key features in second-generation Hadoop.
  • MapReduce: a software framework that allows developers to write programs that process massive amounts of unstructured data in parallel across a distributed cluster of processors or stand-alone computers.
  • Spark: an open source, parallel processing framework that enables users to run large-scale data analytics applications across clustered systems.
  • HBase: a column-oriented key/value data store built to run on top of the Hadoop Distributed File System (HDFS).
  • Hive: an open source data warehouse system for querying and analyzing large data sets stored in Hadoop files.
  • Kafka: a distributed publish/subscribe messaging system designed to replace traditional message brokers.
  • Pig: an open source technology that offers a high-level mechanism for the parallel programming of MapReduce jobs executed on Hadoop clusters.

How big data analytics works

In some cases, Hadoop clusters and NoSQL systems are used primarily as landing pads and staging areas for data before it gets loaded into a data warehouse or analytical database for analysis -- usually in a summarized form that is more conducive to relational structures.

More frequently, however, big data analytics users are adopting the concept of a Hadoop data lake that serves as the primary repository for incoming streams of raw data. In such architectures, data can be analyzed directly in a Hadoop cluster or run through a processing engine like Spark. As in data warehousing, sound data management is a crucial first step in the big data analytics process. Data being stored in the Hadoop Distributed File System must be organized, configured and partitioned properly to get good performance out of both extract, transform and load (ETL) integration jobs and analytical queries. 

Structured and unstructured data can be analyzed using big data analytics.
Big data analytics involves analyzing structured and unstructured data.

Once the data is ready, it can be analyzed with the software commonly used for advanced analytics processes. That includes tools for data mining, which sift through data sets in search of patterns and relationships; predictive analytics, which build models to forecast customer behavior and other future developments; machine learning, which taps algorithms to analyze large data sets; and deep learning, a more advanced offshoot of machine learning.

Text mining and statistical analysis software can also play a role in the big data analytics process, as can mainstream BI software and data visualization tools. For both ETL and analytics applications, queries can be written in MapReduce, with programming languages such as RPython, Scala, and SQL, the standard languages for relational databases that are supported via SQL-on-Hadoop technologies.

Big data analytics uses and challenges

Big data analytics applications often include data from both internal systems and external sources, such as weather data or demographic data on consumers compiled by third-party information services providers. In addition, streaming analytics applications are becoming common in big data environments as users look to perform real-time analytics on data fed into Hadoop systems through stream processing engines, such as Spark, Flink and Storm.

Early big data systems were mostly deployed on premises, particularly in large organizations that collected, organized and analyzed massive amounts of data. But cloud platform vendors, such as Amazon Web Services (AWS) and Microsoft, have made it easier to set up and manage Hadoop clusters in the cloud, as have Hadoop suppliers such as Cloudera and Hortonworks, which support their distributions of the big data framework on the AWS and Microsoft Azure clouds. Users can now spin up clusters in the cloud, run them for as long as they need and then take them offline with usage-based pricing that doesn't require ongoing software licenses.

Potential pitfalls of big data analytics initiatives include a lack of internal analytics skills and the high cost of hiring experienced data scientists and data engineers to fill the gaps.

Recently, the proliferation and advancement of AI and machine learning technologies have enabled vendors to produce software for big data analysis that is easier to use, particularly for the growing citizen data scientist population. Some of the leading vendors in this field include Alteryx, IBM, Microsoft and Knime.

The amount of data that's typically involved, and its variety, can cause data management issues in areas including data quality, consistency and governance. Also, data silos can result from the use of different platforms and data stores in a big data architecture. In addition, integrating Hadoop, Spark and other big data tools into a cohesive architecture that meets an organization's big data analytics needs is a challenging proposition for many IT and analytics teams, which have to identify the right mix of technologies and then put the pieces together.

This was last updated in September 2018

Continue Reading About big data analytics

Join the conversation

21 comments

Send me notifications when other members comment.

Please create a username to comment.

How is "big data" different from "data mining"?
Cancel
Big Data reffers to the full set of information and data mining gathers the techniques you use in order to analyze data in general: big data, small data..
Cancel
would like to know role of intelligent software agents in big data analytics
Cancel
Can anyone start his or her career in data analytics? Whta basics it need?
Cancel
Yes and No.. It all depends on your experience and knowledge in the field. Below is a good article to get a high-level idea on career opportunities in big Data and what each of it takes to enter.
Cancel
At a very high level, Data mining is looking for data based on specifc requests from the client. Big data is analyzing patterns to understand business and create new analytics.
Cancel
Thanks. Great piece. Although the competition has changed during past two years and as mentioned, Hadoop and especially map reduce platforms got much more attention and importance. Due to variety of data sourced and amount of data, players such as tableau, splunk, datameer.com and cloudera getting more and more attention.
Cancel
How could big data help segmenting different customer group needs
Cancel
Having understood what Big Data is all about, can someone please give a list of all the popular Big data software innovators. I have a small list with me which includes Companies like Amazon, IBM etc. What I need is something which is affordable for my company. I've heard of a company called Qburst Technologies which affords to give its customers satisfaction coupled with low pricing.
Cancel
What kind of big data analytics challenges does your organization face? And what are you doing to overcome them?
Cancel
They are many issues an organization face if the implement big data 
Mainly performance issues if system architecture allows optimization then issues can be resoled.

Other issue is with data accuracy and validation?


Cancel
Having gone through several writings on Big data analytics , I am convinced that there are several areas in which it's application in certain areas of our operation could increase our market share and ultimately enhance our bottomline as a bank playing in retail sector 
Cancel
Big data is the most important aspect which all have to be aware of in the field of buisness..
If one want to be in some of the best management companies one must know about all these aspects..
Cancel
To start your career it is a good idea to get familiar with the latest tools after you have a basic understanding. ~ Christopher Gruden, Cleveland, OH
Cancel

Some tools I suggest:

Pentaho
Karmasphere

~ Christopher Gruden, Cleveland, OH

Cancel
And one more: Talend Open Studio

~ Christopher Gruden, Cleveland, OH

Cancel
And Skytree Server. ~ Christopher Gruden, Cleveland, OH
Cancel
How is "big data" different from "data mining"?
Cancel
Our is a company with large amount time series data with milliseconds resolution. The approximate data storage size per day is 150GB. What sort of Big data applications can be used for time series data.
Cancel
What does a data scientist actually do? Can someone explain it with briefly with an example?
Cancel
Hello Sgilan! I suggest you check out our definition for data scientist. It discusses roles, skills, responsibilities and much more. 
Cancel

-ADS BY GOOGLE

File Extensions and File Formats

Powered by:

SearchDataManagement

SearchAWS

SearchContentManagement

SearchCRM

SearchOracle

SearchSAP

SearchSQLServer

SearchSalesforce

Close