Guide to big data analytics tools, trends and best practices
A comprehensive collection of articles, videos and more, hand-picked by our editors
Organizations need to develop data-driven decision-making skills to capitalize on new and emerging big data technologies, attendees were told at this week's MIT Sloan CIO Symposium in Cambridge, Mass. Intuitive decisions are not the path to success when working with the mountains of structured and unstructured information companies are collecting in big data environments, conference presenters said.
"Big data has changed the C-suite entirely," said Shawn Banerji, managing director at Russell Reynolds Associates, a New York-based technology recruiter. "Reliance on empirical data is in. Going with your gut is out."
While computers and computer users continue to create more information and technologists find new ways to slice and dice it, the skills required to analyze the data remain in short supply. It will be challenging for companies to fill leadership positions with individuals who truly understand data and how to use it effectively, MIT researchers said during a panel discussion on "The Reality of Big Data."
Data under the microscope
Big data is a "measurement revolution" that must be accompanied by a similar "management revolution," said Erik Brynjolfsson, a professor at MIT's Sloan School of Management and director of the MIT Center for Digital Business.
Brynjolfsson said changes brought on by greater data availability resemble changes in biology brought about by 15th-century lens maker Anton van Leeuwenhoek's invention of the microscope. Van Leeuwenhoek's early technical letters describing the minute life forms he had seen under the microscope were met with misunderstanding and derision, because many of the established scientists of the day simply could not embrace the idea that microorganisms existed. Today's chief executive officers and chief information officers may be even less ready to believe what the big data in their systems tells them.
For more on big data analytics and data-driven decisions
Learn out about the cognitive bias toward ignoring data
Enter the world of the analytics rock star
"We need to change the decision process from one going with hunches and opinions to one [based on] facts and data," Brynjolfsson said. That means obtaining better data analysis skills, especially in the area of statistical probability.
When correlation met causality
Brynjolfsson warned that unwarranted inferences can be a negative side effect of big data, and said that when unexpected data is encountered, it's a good idea to ask more questions, not less. New kinds of data, much of it semi-structured and unstructured, are often difficult to assess and can be fuzzy. There are whole sets of "weird data" that emerge, Brynjolfsson said. The result can be that inferences made based on the data are untrue.
"Correlation is not causality. This is the most significant mistake people make with data," he said, echoing a common adage teachers use to remind science students that elements that appear together may not actually have a cause-and-effect relationship. Assuming such a relationship can lead to false inferences. The same adage was a prominent theme in The Signal and the Noise, the recent bestseller by big data wunderkind and The New York Times blogger Nate Silver.
Big data has changed the C-suite entirely.
Shawn Banerji, managing director, Russell Reynolds Associates
As one might expect at a university symposium, education was seen as a possible path to better data-driven decision making when working with big data sets. Better education in statistical inference is important, panel members said, even at the high school level.
Front-line big data decisions
While CEOs and CIOs need to prepare for big data decision-making, they should also be ready to delegate decisions to other managers. Often, people on the front lines can make the better decisions, said Frank Diana, a principal consultant at Tata Consultancy Services Ltd.'s U.S. operations.
"It's hard to get people used to not making decisions on a gut level. The key is that decision making has to be pushed closer and closer to the edge, where business is being conducted," said Diana, who took part in another big data panel session.
Diana said the term big data may come and go, but the need for new sets of data analysis skills will only continue to grow. "One of the fundamental elements driving success in the future is analytics excellence," he said.
However, a view contrary to the ''go with the data" approach came by way of a conference attendee who suggested that the big data panel members should consider the beneficial role of intuition. He cited the case of the late Apple CEO Steve Jobs, who would famously shun research data, saying, "People don't know what they want until you show it to them."