There are two very significant ways that our world has changed in the past decade. Firstly, we are more connected. Secondly we are awash with data.
In a planet with 7 billion people there are now 2 billion PCs and upward of 6 billion mobile connections. Besides these connection which we as human beings have, there are now numerous connections to the internet from devices, sensors and actuators. In other words the world is getting more and more instrumented.
There are in excess of 30 billion RFID tags which enable tracking of goods as they move from warehouse, to retail store, sensors on cars and bridges besides cardiac implants in the human body that are constantly sending a stream of data to the network which is known as “the internet of things.” In addition we have the emergence of the Smart Grid with its millions and millions of smart meters that are capable of sensing power loads and appropriately redistributing power and drawing less power during peak hours.
All these devices - be it laptops, cell phones, sensors, RFIDs or smart meters - are sending enormous amounts of data to the network. In other words there is an enormous data overload happening in the networks of today. According to a Cisco report the projected increase in data traffic between 2014 and 2015 is of the order of 200 exabytes. In addition the report states that the total number of devices connected to the network will be twice the world population or around 15 billion.
Fortunately the explosion in data has been accompanied by falling prices in storage and extraordinary increases in processing power. The data that is generated by the devices by the devices, cell phones, PC etc by themselves are useless. However if processed they can provide insights into trends and patterns which can be used to make key decisions.
For example, the data exhaust that comes from a user's browsing trail - also known as the click stream - provide important insights into user behavior, which can be mined to make important decisions. Similarly inputs from social media like Twitter or Facebook provide businesses with key inputs which can be used for making business decisions. Call Detail records that are created for mobile calls can also be a source of user behavior. Data from retail store provide insights into consumer choices. For these to happen the enormous amounts of data has to be analyzed using algorithms to determine statistical trends, patterns and tendencies in the data.