Friday, 5 July 2013

Big Data & Apache Hadoop

Big Data & Apache Hadoop

Introduction

In today's world data is growing at tremendous rate. The Social networking websites, digital photos, Videos, purchases, cell phone all are generating data. Every analysis, study & research is all about data. Its like an outburst of data & it will go on. In future, data would grow in exponential rate. It would increase both the volume and variety of data. Handling this size of data is now becoming out of scope for traditional RDBMS systems. 

Here, comes the Big Data for our rescue.It is being considered as a solution to this data blast.How Big is Big Data? GBs, TBs, Its much more than that PB(that is, Peta Bytes = 1015)
The web definition of Big data says that "Big data is a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications."

Apache Hadoop is an open source tool that help us dive safely in the sea of data and search for the hidden treasure. Hadoop was initially inspired by paper published by Google outlining its approach to handling an avalanche of data. Doug Cutting is the creator of Hadoop and he named it after his son's toy elephant.
It enables the distributed processing of very large data-sets across clusters of commodity hardware. It can reside on a single server or can scale upto 1000s of systems. It is designed to work effectively in the distributed environment.

Major Concepts used in Hadoop

  1. HDFS
  2. Map Reduce

Daemons in Hadoop

  1. NameNode
  2. DataNode
  3. JobTracker
  4. TaskTracker
Reference:
http://en.wikipedia.org/wiki/Big_data
http://developer.yahoo.com/hadoop/tutorial/module1.html