Sergey Nivens - Fotolia
The battle for data supremacy is in full swing. For an organization to be competitive, it's no longer sufficient to sell the best products, produce them as cheaply as possible, handle distribution and delivery expertly, and streamline business processes to optimize efficiency. To survive and thrive, a company also has to manage its data impeccably and exploit the business value of that data to the greatest extent possible. And what may be crucial is whether top management is ready for the fight -- and prepared to invest in big data systems.
Over the past two decades, many data warehouses and business intelligence (BI) systems have been developed. They allow organizations to run reports that, for example, show up-to-date overviews of revenue and detail what customers have bought recently. All of that information is very valuable -- but it's not enough anymore.
In addition to knowing what a customer has purchased, an organization needs to know what he likes, what he's likely to buy, if he comments on the products and the company on social media, whether he's related to other customers, what his demographic characteristics are, and so on.
New rule of engagement: Best data wins
Increasingly, companies will compete on the basis of their data and how they take advantage of it. The organization that has access to the most comprehensive and highest-quality data, that integrates different data sources best, that analyzes data best, and that knows how to use information in the most creative ways will have a big advantage over the competition.
And big data has a big role to play in this battle. It means that more data is available on customers and business processes. And it enriches the ways that organizations can benefit from data.
But to develop effective big data systems, new tools are needed. The technology used in all those BI systems built during the past two decades is limited. More advanced alternatives now allow us to exploit a multitude of different data types. Examples include Hadoop clusters, SQL-based analytical databases, NoSQL database software, in-memory analytics tools and complex event processing engines. Without some or all of them in a big data architecture, it's not going to be possible to gain maximum benefits from the available data.
And having an IT department that understands how a big data environment works isn't enough. It's equally important that the top management of an organization be aware of what's technologically feasible with big data tools and what the limitations are. Not being up to date means not participating in the data battle, or at least not winning it.
IT lessons to learn from real battles
There's a lot to learn from the history of warfare about the importance of having executive leadership that understands technology. I'll cite two examples, the first from the battle of Chancellorsville during the American Civil War, in 1863. At the time, the number of troops available to fight was crucial: The more you had, the greater the chance of winning. As a result, splitting up an army for a battle was a risky strategy, especially because communication technology that could be used to coordinate a combined attack by multiple groups was iffy. Messages often had to be transmitted by dispatch riders on horseback.
But to prepare for the engagement at Chancellorsville, the commanding officer of the Union Army, Maj. Gen. Joseph Hooker, split his forces in two as part of a plan to attack the opposing Confederate States Army from multiple directions. He trusted in a telegraph system to enable the Union forces to communicate over long distances. But the positions of the two army groups were too far apart, and the technology was stressed beyond its capabilities. It didn't work.
Worse, Hooker and his staff were warned in advance that the signal wire used to support the system was untrustworthy. The Union Army's acting chief signal officer, Captain Samuel T. Cushing, reported afterward that he had "requested permission to abandon this line and bring in the wire for repairs, but was refused." The lack of effective communication capabilities contributed to the Union Army losing the battle even though it had twice as many troops as the more agile Confederates did. It demonstrates what can happen when top management doesn't have enough knowledge of technology to make the best choices for an organization.
History has also shown that a country that wins a war often starts its next one using similar techniques and technology. It brought them success the last time, right? Losing countries do the opposite, investing in new technology to try to avoid another defeat (if they have the means to do so). That's what Germany did between the two world wars and what the Allied countries didn't do, leaving them conquered or playing catch-up after World War II began.
The technology being deployed today will play a major role in determining which organizations win the data battle. And a large part of the battle hinges on deploying the right technologies in big data systems. Senior executives need to understand big data processing, storage and analysis technologies and have the vision to invest in them.
About the author:
Rick van der Lans is an independent consultant, speaker and author, specializing in data warehousing, business intelligence, database technology and data virtualization. He is managing director and founder of R20/Consultancy; email him at firstname.lastname@example.org.
Get tips from Rick van der Lans on evaluating and selecting SQL-on-Hadoop engines
Find out how highly networked environments can benefit from distributed data virtualization
Learn why it's time to move query processing closer to the data