September 22, 2012 Leave a comment
An Overview of Big Data
This topic covers my understanding of Big Data and the current ways to exploit its potential.
The phenomenon of Big Data started when companies wanted to store each and every digital interaction generated by their business. The modern technology made things faster, and the availability of processing more orders per minute, gave the companies a faster sales model and the ease of access to a wide variety of products and services.
This fast growth of data let companies record every transaction performed by the customers in different sectors, and the need to have this data was translated into a new necessity . The term Big Data is referred to big volumes of data looking for a storage need.
Depending on the kind of business, it is possible to have a very fast growing data like image processing or image recognition or to have a granular growth of data like strings inserted in a database after a returning customer does the shopping list.
The concept of Data warehouse came into the spotlight when the companies needed to store this old information in places to be analyzed further. Inside the data warehouse, the information started being analyzed using Data mining and a set of new techniques which includes computer learning and forecasting trends.
This analysis was critical to take decisions, understand the business’ behavior , create better marketing strategies as well as advertising campaigns to maximize the complete model.
What’s after the “Big Data”?
Thousands of companies are storing their big amounts of data in data centers but not everybody is processing the information they have got. The next step after the data is gathered and stored is to analyze it through a magnifying glass, if possible.
How to connect the missing parts, how to realize what you don’t know? This question has become the Holy Grail of information these days. As mentioned before, the final purpose of storing these huge volumes of data, was to be analyzed in order to have a more solid base to take proper decisions, to explore new paths, to introduce or remove a service/feature that could be recognized after the in-depth analysis of it.
Is there any problem with Big Data and Data Analysis?
Today we are still computing the same way we used to compute 20 years ago. Processors run on the same binary basis and we have built grids and clouds to maximize the computing power which leads me to mention the first problem of this new approach .
1. Computational complexity – Big amount of data will require thousands of eyes looking at the same project, which is not practical due the volume of data. This has made us develop computer learning techniques to make things easier, but the algorithmic complexity breaks the barriers of what we currently have to process. For instance, new information about the Higgins Boson (The God’s Particle) was recently found, but this data was being analyzed for more than 10 months before it could have any use for science.
2. Business Understanding – More than an algorithm and a group of data miners with the latest tools, what is highly required for the company is a group of people who know about the business. For instance, nobody knows more about a shop than the founder/owner of the same shop; only the vision and requirement of this person who knows the business from A to Z, can grant a better approach towards the decision making.