Hadoop - Bigdata Development
Apache Hadoop is an open-source software framework for storage and large-scale processing of data-sets on clusters of commodity hardware. Hadoop is an Apache top-level project being built and used by a global community of contributors and users. It is licensed under the Apache License 2.0,The Apache Hadoop framework is composed of the following modules:
Hadoop Common – contains libraries and utilities needed by other Hadoop modules
Hadoop Distributed File System (HDFS) – a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster.
Hadoop YARN – a resource-management platform responsible for managing compute resources in clusters and using them for scheduling of users' applications.
Hadoop Map Reduce – a programming model for large scale data processing.
All the modules in Hadoop are designed with a fundamental assumption that hardware failures (of individual machines or racks of machines) are common and thus should be automatically handled in software by the framework. Apache Hadoop's Map Reduce and HDFS components originally derived respectively from Google's Map Reduce and Google File System (GFS) papers.
Basics of Hadoop: Motivation for Hadoop Large scale system training Survey of data storage literature Literature survey of data processing Networking constraints New approach requirements Basic concepts of Hadoop What is Hadoop? Distributed file system of Hadoop Map reduction of Hadoop works Hadoop cluster and its anatomy Hadoop demons Master demons Name node Tracking of job Secondary node detection Slave daemons Tracking of task HDFS(Hadoop Distributed File System) Spilts and blocks Input Spilts HDFS spilts Replication of data Awareness of Hadoop racking High availably of data Block placement and cluster architecture CASE STUDIES Practices & Tuning of performances Development of mass reduce programs Local mode Running without HDFS Pseudo-distributed mode All daemons running in a single mode Fully distributed mode Dedicated nodes and daemon running Hadoop administration Setup of Hadoop cluster of Cloud era, Apache, Green plum, Horton works On a single desktop, make a full cluster of a Hadoop setup. Configure and Install Apache Hadoop on a multi node cluster. In a distributed mode, configure and install Cloud era distribution. In a fully distributed mode, configure and install Hortom works distribution In a fully distributed mode, configure the Green Plum distribution. Monitor the cluster Get used to the management console of Horton works and Cloud era. Name the node in a safe mode Data backup. Case studies Monitoring of clusters Hadoop Development : Writing a MapReduce Program Sample the mapreduce program. API concepts and their basics Driver code Mapper Reducer Hadoop AVI streaming Performing several Hadoop jobs Configuring close methods Sequencing of files Record reading Record writer Reporter and its role Counters Output collection Assessing HDFS Tool runner Use of distributed CACHE Several MapReduce jobs (In Detailed) 1.MOST EFFECTIVE SEARCH USING MAPREDUCE 2.GENERATING THE RECOMMENDATIONS USING MAPREDUCE 3.PROCESSING THE LOG FILES USING MAPREDUCE Identification of mapper Identification of reducer Exploring the problems using this application Debugging the MapReduce Programs MR unit testing Logging Debugging strategies Advanced MapReduce Programming Secondary sort Output and input format customization Mapreduce joins Monitoring & debugging on a Production Cluster Counters Skipping Bad Records Running the local mode MapReduce performance tuning Reduction network traffic by combiner Partitioners Reducing of input data Using Compression Reusing the JVM Running speculative execution Performance Aspects CASE STUDIES CDH4 Enhancements : 1. Name Node – Availability 2. Name Node federation 3. Fencing 4. MapReduce – 2 HADOOP ANALYST 1. Concepts of Hive 2. Hive and its architecture 3. Install and configure hive on cluster 4. Type of tables in hive 5. Functions of Hive library 6. Buckets 7. Partitions 8. Joins 1. Inner joins 2. Outer Joins 9. Hive UDF PIG 1. Pig basics 2. Install and configure PIG 3. Functions of PIG Library 4. Pig Vs Hive 5. Writing of sample Pig Latin scripts 6. Modes of running 1. Grunt shell 2. Java program 7. PIG UDFs 8. Macros of Pig 9. Debugging the PIG IMPALA 1. Difference between Pig and Impala Hive 2. Does Impala give good performance? 3. Exclusive features 4. Impala and its Challenges 5. Use cases NOSQL 1. HBase 2. HBase concepts 3. HBase architecture 4. Basics of HBase 5. Server architecture 6. File storage architecture 7. Column access 8. Scans 9. HBase cases 10. Installation and configuration of HBase on a multi node 11. Create database, Develop and run sample applications 12. Access data stored in HBase using clients like Python, Java and Pearl 13. Map Reduce client 14. HBase and Hive Integration 15. HBase administration tasks 16. Defining Schema and its basic operations. 17. Cassandra Basics 18. MongoDB Basics Ecosystem Components 1. Sqoop 2. Configure and Install Sqoop 3. Connecting RDBMS 4. Installation of Mysql 5. Importing the data from Oracle/Mysql to hive 6. Exporting the data to Oracle/Mysql 7. Internal mechanism Oozie 1. Oozie and its architecture 2. XML file 3. Install and configuring Apache 4. Specifying the Work flow 5. Action nodes 6. Control nodes 7. Job coordinator Avro, Scribe, Flume, Chukwa, Thrift 1. Concepts of Flume and Chukwa 2. Use cases of Scribe, Thrift and Avro 3. Installation and configuration of flume 4. Creation of a sample application Challenges of Hadoop 1. Hadoop recovery 2. Hadoop suitable cases.