Technology will largely solve big data skills gap

Yet another report, this one from non-profit TechAmerica Foundation, says there are hundreds of job vacancies in the U.S. for data scientists but a dearth of people with the right skills to fill these jobs. The report was released this week and follows similar studies from CompTIA and the now infamous report by McKinsey which estimated a shortage of 200,000 skilled data managers in the U.S. by 2016.

This skills issue is getting overblown in my opinion and here’s why.  My calendar has been booked solid with briefings from startups building next generation data warehouses and BI tools. I believe that businesses willing to test these new products will be pleasantly surprised at the leap in ease of use, cost and performance of newer data warehouses, a side benefit of which is not having to recruit highly skilled, expensive data scientists.

But hang on just a minute you say. We spent millions of dollars on the current generation of data warehouse products and look where that got us? One-off, custom-built Frankensteins with complicated architectures that took specialist expertise to operate. How’s it going to be any different this time around?

Summarizing all the briefings I’ve had in the past week or so, here’s the promise:

• Lots of automated ways of getting data into the warehouse
• Highly customizable on the front-end with different apps (think iPhone) and as simple to operate on the back-end
• Fast setup and time to answers
• Eco-system, partner orientated (vendor neutral, standards neutral)
• Massively scaleable (interactive queries against petabytes of data in hours, not months)
• Choice of cloud or on premise solutions
• 1: 10 reduction in cost due to automation of the transport/integration layer and commodity hardware

For those willing to try something different, new companies in the data warehouse space aiming to democratize data science include: Treasure Data, Platfora, Qubole, and Precog among others. There are also numerous startups building next-gen BI tools including DataHero, PriorKnowledge, ClearStory, and BigML. Check them out. I’d be curious to hear from anyone that still thinks we need to panic about the shortage of data scientists after using these products.