Thursday, 10 December 2015

Faster way to count number of lines in a file/dir using Map Reduce Framework


In this site you can see one way to count number of lines in a file.
They are emitting count as one for each record in each map. So if 1 map holds 10,000 lines 10,000 values will be passed to reducer.If more than one mapper that many read-writes will happen.
Lets reduce the intermediate writes.

Below is an optimized way to count no of lines in a file/dir
Changes are done in
1. Mapper
Instead of emitting 'one' for each record, we increment line count in map and emit them in cleanup() phase.
public class LineCntMapper extends
  Mapper<LongWritable, Text, Text, IntWritable> {

 Text keyEmit = new Text("Total Lines");
 IntWritable valEmit = new IntWritable();
 int partialSum = 0;

 public void map(LongWritable key, Text value, Context context) {
  partialSum++;
 }

 public void cleanup(Context context) {
  valEmit.set(partialSum);
  try {
   context.write(keyEmit, valEmit);
  } catch (IOException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
   System.exit(0);
  } catch (InterruptedException e) {
   // TODO Auto-generated catch block
   e.printStackTrace();
   System.exit(0);
  }
 }
}
So if we have 5 map tasks we will only emit 5 intermediate key-value pair.

2. Driver
In Driver we will include a combiner also
job.setMapperClass(LineCntMapper.class);
job.setCombinerClass(LineCntReducer.class);
job.setReducerClass(LineCntReducer.class);
Combiner doesnt do nothing more than Reducer. we can use reducer as combiner itself.
Reducer doesnt need any change.

If you run this code you will get the results faster than the previous mentioned code in this site .

Working code is here

Happy Hadooping........

27 comments:

  1. For latest and updated Cloudera certification dumps in PDF format contact us at completeexamcollection@gmail.com.
    Refer our blog for more details http://completeexamcollection.blogspot.in/2015/04/cloudera-hadoop-certification-dumps.html

    ReplyDelete
  2. Hi Sreeveni,
    can you please share the list of interview questions or questions and answers kind of stuff if you have. my mail id is guresh.amara at gmail.com

    ReplyDelete
  3. Your blog is more informative..Its very interesting to know about new

    things..Keep on sharing.
    SEO training in chennai

    ReplyDelete
  4. Wow amazing i saw the article with execution models you had posted. It was such informative. Really its a wonderful article. Thank you for sharing and please keep update like this type of article because i want to learn more relevant to this topic.

    Hadoop Training in Chennai

    ReplyDelete
  5. There are lots of information about latest technology and how to get trained in them, like this have spread around the web, but this is a unique one according to me. The strategy you have updated here will make me to get trained in future technologies. By the way you are running a great blog. Thanks for sharing this.
    Hadoop training in chennai

    ReplyDelete
  6. This article describes the Hadoop Software, All the modules in Hadoop are designed with a fundamental assumption that hardware failures are common and should be automatically handled by the framework. This post gives great idea on Hadoop Certification for beginners. Also find best Hadoop Online Training in your locality at StaygreenAcademy.com

    ReplyDelete

  7. What an awesome post, I just read it from start to end. Learned something new after a long time.


    SAP FICO Training in Chennai

    SAP SD training in Chennai

    ReplyDelete
  8. wow amazing post.The key points you mentioned here related to maintenance of car is really awesome.Checking all fluid levels,changing oil and of course the regular service of the car which is necessary to maintain our vehicle.Thank you for the information.

    home spa services in mumbai

    ReplyDelete
  9. Being new to the blogging world I feel like there is still so much to learn. Your tips helped to clarify a few things for me as well as giving..

    Manpower Consultancy in Chennai

    ReplyDelete
  10. I just see the post i am so happy the post of information's.So I have really enjoyed and reading your blogs for these posts.Any way I’ll be subscribing to your feed and I hope you post again soon.
    best GMAT coaching classes in chennai

    ReplyDelete
  11. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.
    Isoft Innovations Facebook

    ReplyDelete
  12. Finding the time and actual effort to create a superb article like this is great thing. I’ll learn many new stuff right here! Good luck for the next post buddy..
    SEO Company in Chennai

    ReplyDelete
  13. Such a great articles in my carrier, It's wonderful commands like easiest understand words of knowledge in information's.

    Aws Training in Chennai

    ReplyDelete
  14. Great post! I am actually getting ready to across this information, It's very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.

    Seo Company in Chennai

    ReplyDelete

  15. Thanks for this blog. provided great information. All the details are explained clearly with the great explanation. Thanks for this wonderful blog. Step by step processes execution are given clearly.Know the details about different thing.
    Seo Company in India

    ReplyDelete
  16. Finding the time and actual effort to create a superb article like this is great thing. I’ll learn many new stuff right here! Good luck for the next post buddy..
    Fresher Jobs
    Fresher Opening

    ReplyDelete
  17. This is extremely helpful info!! Very good work. It is very interesting to learn and easy to understood. Thank you for giving information. Please let us know and more information get post to link.

    Digital Marketing Training in Chennai

    ReplyDelete
  18. Great post! I am see the great contents and step by step read really nice information.I am gather this concepts and more information. It's helpful for me my friend. Also great blog here with all of the valuable information you have.
    House Cleaning Service in Chennai


    ReplyDelete

  19. تتعدد الشركات التي تقدم خدمات ىالتنظيف لاكن لا يمكن ان تكون كلها في نفس مستوي الجوده فان كنت من الباحثين عن جودة الشركه قبل اي شئ اخر فانصحة بزيارة احدي تلك الصفحات
    شركة تنظيف مساجد بالرياض
    شركة تنظيف خزانات بالخرج
    شركة تنظيف بالخرج
    والتي تقدم افضل خدمات التنظيف بالمنزل باعلي مستوي من الكفائه
    شركة تنظيف منازل بالطائف

    ReplyDelete
  20. This blog is having the general information. Got a creative work and this is very different one.We have to develop our creativity mind.This blog helps for this. Thank you for this blog. This is very interesting and useful.
    Housekeeping Services in Chennai

    ReplyDelete
  21. It's like you read my mind! You seem to know a lot about this, like you wrote the book in it or something. I think that you can do with some pics to drive the message home a little bit, but instead of that, this is fantastic blog. A great read. I will definitely be back.
    Office Interiors in Chennai
    Interior Decorators in Chennai

    ReplyDelete
  22. Interesting blog which attracted me more.this blog shows that you have a great future as a content writer.keep updating.
    Digital marketing company in Chennai

    ReplyDelete
  23. There are many interesting information included and i can easily understand all given information.I post something on my blog to post something, or wait to post something worth saying. Keep update more information....
    Digital Marketing Training in Chennai
    Web Designing Training in Chennai

    ReplyDelete
  24. Really Good blog post.provided a helpful information.I hope that you will post more updates like this.
    MSBI Training in Chennai

    ReplyDelete
  25. you are posting a good information for people and keep maintin it and give more updates too.
    seo company india

    ReplyDelete
  26. Interesting blog post.This blog shows that you have a great future as a content writer.waiting for more updates...
    seo company in Chennai

    ReplyDelete