What is Hadoop Streaming and Why Should I use it?

Hadoop, formally called Apache Hadoop, is an Apache Software Foundation project and open source software platform for scalable, distributed computing. Hadoop can provide fast and reliable analysis of both structured data and unstructured data. 

Hadoop can scale up from single servers to thousands of machines, each offering local computation and storage.

Hadoop consists of HDFS and MapReduce

HDFS uses a master/slave architecture in which one device (the master) controls one or more other devices (the slaves).

MapReduce takes care of scheduling tasks, monitoring them and re-executing any failed tasks. 


Even though the Hadoop framework is written in Java, programs for Hadoop need not to be coded in Java but can also be developed in other languages like Python or C++. You can translate your Python code using Jython into a Java jar file but this is not convenient.


Or you could write the code in python itself. The "trick" behind the following Python code is that we will use the Hadoop Streaming API

Test your mapper.py and reducer.py scripts locally before using them in a MapReduce job. Before we run the actual MapReduce job, we must first copy the files from our local file system to Hadoop's HDFS.



Besides streaming you can also use PyDoop which is a wrapper over Streaming APIs written in Python.

Comments

Post a Comment

Popular posts from this blog

What is Skylake and how does it compare with Broadwell or Haswell?

Haswell, Broadwell and Skylake are are codenames for the respective 4th, 5th and 6th Generation iterations of Intel's processor microarchitechture. Broadwell is perceived as a version upgrade from Haswell however Sklyake is a major microarchitecture re-design. Apple has introduced the latest lineup of Laptops that now use Intel Core M processors based on Skylake. Core i5/i7 are faster than Core M. Core M lies in between the ATOM and the Core i5/i7 series but on a Skylake Architecture however. Intel has the Core i5/i7 versions on both Skylake and Broadwell.
Read more

What is Tokenization?

GTE or Global Tokenization Exchange is going to have a big impact in future The advantages of tokenization are big. Here are a few... Illiquid Assets like Real Estate becomes Liquid Overnight Efficiency in moving around assets as everything is on the blockchain Transparency of assets Fractional Ownership Global Access instead of local Sent from Mailspring , the best free email app for work
Read more

What is three in a box model?

IT Companies in India are now having one of the three senior executives designated as a 'client partner' and there will also be a 'delivery partner' based in India to ensure that work commitments are adhered to. In addition they will have a consulting partner as well.   "The key objective of the consulting partner programme is to help provide greater strategic value to the client organisation by proactively solving clients' business issues, demonstrating domain or functional knowledge and innovation, delivering higher-value strategic and transformational projects," R Chandrasekaran, group chief executive for technology and operations at Cognizant, wrote in an email.   The full article is at http://articles.economictimes.indiatimes.com/2012-09-21/news/34002564_1_infosys-clients-information-technology-bellwether
Read more