Sunday, 4 May 2014

Map-Only Jobs In Hadoop


There may be reasons where Map-Only job is needed,Where there is no Reducer to execute.Here Map does all its task with its InputSplit and no job for Reducer.This can be achieved by setting  job.setNumReduceTasks()  to Zero in Configuration.

Job job = new Job(getConf(), "Map-Only Job");
job.setJarByClass(MaponlyDriver.class);

job.setMapOutputKeyClass(LongWritable.class);
job.setMapOutputValueClass(Text.class);

job.setOutputKeyClass(IntWritable.class);
job.setOutputValueClass(Text.class);
/*
 * Set no of reducers to 0
 */
job.setNumReduceTasks(0);

job.setMapperClass(Mapper.class);

job.setInputFormatClass(TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);

FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));

boolean success = job.waitForCompletion(true);
return(success ? 0 : 1);

This sets Reducer task to 0 and turns off the Reducer.

job.setNumReduceTasks(0);

So the no. of output files will be equal to no. of mappers and output files will be named as part-m-00000.

And once Reducer task is set to Zero the result will be unsorted.

If we are not specifying this property in Configuration, an Identity Reducer will get executed in which the same value is simply emitted along with the incoming key and the output file will be part-r-00000.



Happy Hadooping ...

71 comments:

  1. In what types of situations might it be ideal not to run any reducers?

    ReplyDelete
    Replies
    1. For image processing and all we dont need to run a reducer and even in some preprocessor techniques in dataMining while chaining different jobs there are situation where we dont need a key(Nullwritable key) but only values. In that situation if we dont want output in a single file we can run the job without reducers.

      Delete
  2. Hi this is vignesh i am having 3 years of experience as a android developer and i am certified. i have knowledge on OOPS concepts in android but dont know indepth. After learning hadoop will be enough to get a good career in IT with good package? and i crossed hadoop training in chennai website where someone please help me to identity the syllabus covers everything or not??

    Thanks,
    vignesh

    ReplyDelete
    Replies
    1. Hi Vignesh, I have gone through the training link.It covers almost everything.When did you appeared for the training.If you have confidence you can register for Cloudera certification exam.It is quite easy but tricky.And you will get a good career too. Pls have a look at this: " http://unmeshasreeveni.blogspot.in/2015/01/cloudera-certified-hadoop-developer-ccd.html ". Let me know if you have further queries

      Delete
  3. There are lots of information about latest technology and how to get trained in them, like Hadoop Training Chennai have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies(Hadoop Training in Chennai). By the way you are running a great blog. Thanks for sharing this.

    ReplyDelete
  4. The expansion of internet and other business intelligence leads to large volume of data. Industries are looking for talented professionals to maintain and process huge volume of data with latest tools available in the market. Taking Hadoop Training in Chennai | Big Data Training in Chennai will ensure better career prospects for talented professionals.

    ReplyDelete
  5. Thank you so much.your blog is very helpful.Could you please post any material on Web services testing too.

    Hadoop Certification in Chennai

    ReplyDelete
  6. Thank you so much.your blog is very helpful.Could you please post any material on Web services testing too.
    qlikview training in chennai

    ReplyDelete
  7. I was getting errors trying to run this program with my own input, so I changed "wiki/in" in the driver file to args[0], and it ran successfully.
    But my output seems to match with the sample output of job 1 for each iteration, instead of converting to the formats of job 2 and 3. Could this small change in code have changed so much? How?

    hadoop training in chennai

    ReplyDelete
  8. This is extremely helpful info!! Very good work. It is very interesting to learn and easy to understood. Thank you for giving information. Please let us know and more information get a post to link. Data Science Training | Online Data Science Training





    ReplyDelete
  9. Excellent and very cool idea and the subject at the top of magnificence and I am happy to this post..Interesting post! Thanks for writing it.What's wrong with this kind of post exactly? It follows your previous guideline for post length as well as clarity..
    Android Training in Chennai

    ReplyDelete
  10. This is excellent information. It is amazing and wonderful to visit your site.Thanks for sharing this information&its very useful to me...
    Android training in chennai
    Ios training in chennai

    ReplyDelete
  11. Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.
    Hadoop Training in Chennai

    ReplyDelete
  12. Great and interesting article to read.. i Gathered more useful and new information from this article.thanks a lot for sharing this article to us..

    best institute for big data in Chennai | big data training in Velachery

    ReplyDelete
  13. Useful article.. This article helpful to me for update my hadoop skills ..

    hadoop training in chennai | big data training in chennai

    ReplyDelete
  14. Thank you for taking the time and sharing this information with us. It was indeed very helpful and insightful while being straight forward and to the point.
    mcdonaldsgutscheine.net | startlr.com | saludlimpia.com

    ReplyDelete
  15. Really useful information about hadoop, i have to know information about hadoop online training institutes.

    ReplyDelete
  16. Really Superb Information About hadoop here is the best for Hadoop Training

    ReplyDelete
  17. Excellent post ..really useful blog... thanks for sharing...

    Hadoop Common Training

    ReplyDelete
  18. And indeed, I’m just always astounded concerning the remarkable things served by you. Some four facts on this page are undeniably the most effective I’ve had.
    Hadoop Training Institute In chennai

    amazon-web-services-training-in-bangalore

    ReplyDelete
  19. I simply wanted to write down a quick word to say thanks to you for those wonderful tips and hints you are showing on this site.
    Best Python training Institute in chennai

    ReplyDelete
  20. Great article. Thanks for providing such a wonderful and very useful blog over here.I believe there are many more pleasurable opportunities ahead for individuals that looked at your site.
    selenium training in chennai

    ReplyDelete
  21. I simply wanted to write down a quick word to say thanks to you for those wonderful tips and hints you are showing on this site.
    Best selenium training Institute in chennai

    ReplyDelete
  22. Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offersAWS training in Bangaloreusing classroom and AWS Online Training globally.

    ReplyDelete
  23. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.
    Data Science Training in Chennai
    Data science training in bangalore
    Data science online training
    Data science training in pune
    Data science training in kalyan nagar
    Data science training in Bangalore
    Data science training in tambaram

    ReplyDelete
  24. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.
    Devops training in velachry
    Devops training in OMR
    Deops training in annanagar
    Devops training in chennai
    Devops training in marathahalli
    Devops training in rajajinagar
    Devops training in BTM Layout


    ReplyDelete
  25. I wish to show thanks to you just for bailing me out of this particular trouble.As a result of checking through the net and meeting techniques that were not productive, I thought my life was done.


    ccna training in chennai



    ccna training in bangalore


    ccna training in pune

    ReplyDelete
  26. Thank you so much for a well written, easy to understand article on this. It can get really confusing when trying to explain it – but you did a great job. Thank you!

    java training in annanagar | java training in chennai

    java training in marathahalli | java training in btm layout

    java training in rajaji nagar | java training in jayanagar

    ReplyDelete



  27. Very good brief and this post helped me alot. Say thank you I searching for your facts. Thanks for sharing with us!

    angularjs-Training in velachery

    angularjs-Training in annanagar

    angularjs Training in chennai

    angularjs Training in chennai

    ReplyDelete
  28. Hello! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us useful information to work on. You have done an outstanding job.


    AWS Training in Rajaji Nagar | Amazon Web Services Training in Rajaji Nagar


    Amazon Web Services Training in Pune | Best AWS Training in Pune

    AWS Online Training | Online AWS Certification Course - Gangboard

    ReplyDelete
  29. I likable the posts and offbeat format you've got here! I’d wish many thanks for sharing your expertise and also the time it took to post!!
    python training in velachery
    python training institute in chennai

    ReplyDelete
  30. Great post! I am actually getting ready to across this information, It’s very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.

    Devops Training in pune
    DevOps online Training

    ReplyDelete
  31. Read all the information that i've given in above article. It'll give u the whole idea about it.
    Selenium Training in Chennai | Selenium Training in Bangalore |Selenium Training in Pune | Selenium online Training

    ReplyDelete
  32. Amazon Web Services (AWS) is the most popular and most widely used Infrastructure as a Service (IaaS) cloud in the world.AWS has four core feature buckets—Compute, Storage & Content Delivery, Databases, and Networking. At a high level, you can control all of these with extensive administrative controls accessible via a secure Web client.For more information visit aws online
    training

    ReplyDelete
  33. Thank you for your guide to with upgrade information about Hadoop
    Hadoop admin Online Course

    ReplyDelete
  34. Thanks for sharing steps. This is really helpful. Keep doing more.

    easyblogging
    Guest posting sites

    ReplyDelete
  35. I love the blog. Great post. It is very true, people must learn how to learn before they can learn. lol i know it sounds funny but its very true. . .
    python training in chennai
    python course in chennai
    python training in bangalore

    ReplyDelete
  36. Some us know all relating to the compelling medium you present powerful steps on this blog and therefore strongly encourage contribution from other ones on this subject while our own child is truly discovering a great deal. Have fun with the remaining portion of the year.

    Selenium training in Chennai | Selenium training institute in Chennai | Selenium course in Chennai

    Selenium training in Bangalore | Selenium training institute in Bangalore | Selenium course in Bangalore

    Selenium interview questions and answers

    Selenium training in Pune | Selenium training institute in Pune | Selenium course in Pune

    ReplyDelete
  37. I found your blog while searching for the updates, I am happy to be here. Very useful content and also easily understandable providing.. Believe me I did wrote an post about tutorials for beginners with reference of your blog.

    rpa training in chennai |rpa course in chennai|
    rpa training in bangalore | best rpa training in bangalore | rpa course in bangalore | rpa training institute in bangalore | rpa training in bangalore | rpa online training

    ReplyDelete
  38. Thank you for allowing me to read it, welcome to the next in a recent article. And thanks for sharing the nice article, keep posting or updating news article.
    Best Devops online Training
    Online DevOps Certification Course - Gangboard

    ReplyDelete