Few topics have lately drawn more hype and scrutiny than Big Data. Having originated with Internet firms, Big Data has captured enterprise attention with examples that show how organizations, from public sector to financial services firms, telcos, and media derived insights that improved customer retention, operational efficiency, and risk mitigation.
Big Data has drawn media attention with examples that touch people’s lives, such as the massive data hunt that occurred in the wake of the Boston Marathon bombing. Yet few technology trends have also drawn so many myths – such as that Big Data is a special project that can only be undertaken by uniquely skilled professionals, requiring its own dedicated infrastructure. Although that was the model for early Big Data projects among Internet firms, that model is not sustainable for the enterprise. To hit the enterprise mainstream, Big Data must become a first-class citizen with IT, the data center, and the enterprise.
SWAT team origins of Big Data
Big Data – as defined by huge volume, wide variety, and in some cases, extreme velocity – overwhelmed traditional enterprise databases. SQL databases required that data be modeled ahead of time based on expected query patterns. That model broke down for Internet companies seeking to build search indexes, optimize ad placement, or enhance online gamer experiences. As a result, Internet firms created their own technology, open-sourced it, and required special expertise and dedicated infrastructure to run Big Data (primarily, but not exclusively Hadoop). There were few concerns over security, capacity utilization, data stewardship, or information lifecycle management, as the stakes for market dominance were high and resources deep.
This model is clearly unsustainable for mainstream enterprises, which cannot afford to replace their SQL developers with new talent; run Hadoop clusters as separate islands; or treat every question as a unique data science exercise. Enterprises are clearly interested in tapping Big Data, but to do so, Big Data systems must become accessible to the existing IT organization bank of talent, while requiring some acquisition of new skills; respect existing data center policies, practices, and constraints; and address familiar enterprise business issues.