Business Information

Technology insights for the data-driven enterprise


Shortage of data scientists, big data pros vexes IoT efforts

There's an abundance of big data technologies that can support Internet of Things projects, but the pool of IT professionals with the advanced know-how to handle massive amounts of IoT data is limited.

It may be the Internet of Things, but it takes people to make IoT data management and analytics projects work. The problem is there's a shortage of data scientists and data management professionals with skills in using Hadoop and other big data technologies -- at least at salary levels that won't bust budgets.

Joe Hsy, director of the cloud services platform and tools at Cisco Systems Inc.'s WebEx conferencing unit, said the competition in the job market is intense on both the processing and analytics sides. WebEx is using Hadoop, Spark, Cassandra and other in-demand technologies to collect and analyze data from the mobile devices and PCs of people using its Web and video conferencing services. To help ensure that his team can use those technologies effectively, Hsy said he is working to expand the skills of existing employees, while "aggressively hiring" new workers when he finds them.

Lack of big data scientists, preventing growth for IOT.

The shortage of data scientists and other workers with big data skills was one of the reasons Automatic Labs Inc. decided to adopt Databricks' cloud-based Spark platform, a change in strategy after the San Francisco company initially began doing its own development with the base Apache Spark open source software. "It's very difficult to find infrastructure engineers in San Francisco," said Rob Ferguson, vice president of software engineering at Automatic Labs. "And the cost is just tremendous -- probably $200,000 a year per person."

It's the same at Rockwell Automation Inc., which used Microsoft's Azure cloud technologies to build a platform for storing and analyzing data captured from manufacturing equipment at companies that use its industrial automation tools. "We don't want to be building infrastructure that's plumbing," said John Dyck, Rockwell's global director of software business development. "We want to focus on manufacturing functionality and analytics because that's our value-add."

Article 5 of 10

Next Steps

The rundown on cloud service management and cloud monitoring tools

Analyzing big data in the cloud

A guide to understanding IoT data

Dig Deeper on Big data analytics

Get More Business Information

Access to all of our back issues View All