Data-related investments shift from technology to individual skills – talent the new differentiator

Over the last decade, the access to best-of-bread data technologies has become easier. This is due mainly to the increasing popularity of open source software (OSS). While this phenomenon holds true in other areas like operating systems, application servers, development frameworks or even monitoring tools, it is perhaps most prevalent in the area of data.

Not many people will argue with the fundamental role OSS plays in Big Data or NoSQL. These categories were virtually built on OSS. Just consider Hadoop, MongoDB and Redis as a small sample. What surprised me though is how these solutions have infiltrated the enterprise market and caused a drastic shift in the way organizations are now spending their money on data-related projects.

Funds allocation for data related projects

This shift away from technology and investment into skills is primarily due to two overarching technology trends:

• Adoption of Cloud services as the predominate means of delivering applications; not software
• Prevalent lack of support for scale-out architecture in traditional data solutions and its dependency on proprietary hardware

Just consider this; today, whether I work for a small start-up or a large conglomerate, I have access to the same data technology that is used by the most popular companies in the world: Twitter, Facebook, Instagram just to name a few. This new technologies are have been designed from ground up to meet the scale of on-line applications and are also available as open source. However, the issue is that many of these data solutions meet one and only one specific issue. It is the skills, expertise and endless hours of experts that make these data products truly valuable to organizations. Unfortunately, in the hands of many untrained professionals, these powerful tools, while free, are useless!

There are already some high-profile examples of this shift to OSS-based data solutions in the enterprise. Disney, which recently announced their Data Management Platform, has built it on a “start-up-like” budget. This complex system uses Hadoop and NoSQL databases, coupled with some innovative API architecture that shields developers from product complexities and enables standard approaches to data management, regardless of its format.

Sears also announced last week that it was going “all-in on Hadoop”. They found their traditional (proprietary) data solution not flexible enough, and decided that to remain relevant, this old-school company had to adopt to new (Big Data) technologies. Sears is in fact so committed to this new direction, that they are actually planning on re-selling their new Big Data solution as a service.

These two Fortune 100 companies are but a precursor to what we will see over the next five years as a result of this growing trend in today’s enterprise. While everyone may have access to free analytical solutions like Hadoop, open source key-value like Redis or document-based database like MongoDB, the skill and expertise needed to glue these products into differentiated solutions is hard to come by.

This is where I can see another shift that is currently impacting the popularity of OSS in the enterprise: organizations are increasingly willing to work with multiple, best-of-breed, vendors.

Willingness to work with multiple best-of-breed vendors

The traditional desire to have a single partner responsible for all issues appears to have been overwritten by competitive advantages and the fear of vendor lock-in. The money saved by making these OSS choices gives now organizations the luxury of outsourcing to a variety of technology experts.

For example, when Disney needs support with a Hadoop cluster, they call Cloudera, When they have questions about Solr or Cassandra implementation they bring in DataStax. The interesting part of this trend is that it even applies to propitiatory products that leverage OSS as in case of Sears leveraging Datameer’s expertise in analytical tools.

I’m sure there are many examples of where this kind of best-of-breed approach to data delivered less than optimal results. But, to remain competitive enterprise has to make systems that scale and are able to quickly adopt to the ever-changing application demands. Right now, these systems seem to be built on top of data platforms based on open source. And, since the access to these technologies has been commoditized, it is the skill of personnel that’s becoming the true differentiator.

Unfortunately, in many cases, companies shy away from training their people in these solutions, fearing they will leave for greener pastures. That’s however a very shortsighted perspective. They really should fear what will happen if they don’t train them and they stay!