The stories you hear about what can be accomplished with big data can be truly mind-boggling. The real-time stats package Major League Baseball generates with each pitch and at-bat elicited the biggest oohs and aahs at the AWS Re:Invent conference late last year. Super accurate weather and soil condition forecasts are helping Georgia farmers optimize irrigation plans.
Whatever the reason for running or experimenting with big data, the same filters must be applied to big data as to any big IT project: What's the business problem to be solved? Can big data technology solve it? Are we analyzing the right (or right amount) of data?
The consequences can be overwhelming. As Harvard Business School professor Clayton Christensen says in the cover story in this issue of Business Information, "Big data for big data's sake just gives us more data, and that's not the insight I think we need.”
At Allstate Insurance Co., quantitative research and analytics fellow Mark Slusar found the right mix in a Hadoop data lake: "Previously, a lot of the data we looked at was only at the state level because data at the country-wide level was so large that we didn't have an effective way to work with it," he says. "Now it's more organized and centrally located, and the computing power is leaps and bounds faster than before. What used to take months now takes hours," Analyzing that data enabled Allstate to reduce the number of property inspections it does, saving it about $3 million in 2014.
Still, big data projects should not be approached without careful thought. The investment in technology needs to be matched by investment in human expertise -- to define the problems and data sets, to ask the right questions about the data and to pull the relevant information from the data.
Speaking of expertise, I'd like to welcome a new columnist to the Business Information lineup: Celso Mello, CIO of Reliance Home Comfort, a supplier of home heating and cooling systems in Canada. In each issue he will bring an IT executive perspective to solving business problems with technology. His message in this issue, about how to become a CIO, even applies to the problem of big data. "A high-profile project can transform a business, so it requires executive involvement," he writes.
Good advice for anyone who understands the value of technology.
Are you venturing into the big data landscape? Write me at email@example.com.
'Big data' analytics projects are easier said than done
Big data projects are driving health IT investments
Big data analytics projects raise the stakes for predictive models