Thursday, 10 December 2015

DoubleArrayWritable in Hadoop


Lets see how to emit double arrays from mapper and process them in reducer

DoubleArrayWritable class
public static class DoubleArrayWritable extends ArrayWritable {
 public DoubleArrayWritable() {
  super(DoubleWritable.class);
 }
}

Driver()
job.setMapOutputKeyClass(IntWritable.class);
job.setMapOutputValueClass(DoubleArrayWritable.class);

job.setOutputKeyClass(NullWritable.class);
job.setOutputValueClass(DoubleArrayWritable.class);

map()
import mywritable.DoubleArrayWritable;

public class MyMapper extends
Mapper<Object, Text, IntWritable, DoubleArrayWritable> {

 public void map(Object key, Text value, Context context)
 {
  //Do something............
  double[] arr = new double[size];

  DoubleArrayWritable arrWritable = new DoubleArrayWritable();
  DoubleWritable[] data = new DoubleWritable[size];
  for (int k = 0; k < size; k++) {
   data[k] = new DoubleWritable(arr[k]);
  }
  arrWritable.set(data);

   context.write(mykey, arrWritable);
 }
}

reduce()
import mywritable.DoubleArrayWritable;

public class MyReducer  extends
Reducer<IntWritable, DoubleArrayWritable, NullWritable, Text> {

 public void reduce(IntWritable key,
   Iterable<DoubleArrayWritable> values, Context context){

  double[] sum = new double[size];
  for (DoubleArrayWritable c : values) {
    Writable[] temp = new DoubleWritable[size];
    temp = (c.get());
    for (int i = 0; i < size; i++) {
         sum[i] += Double.parseDouble(temp[i].toString());
    }

   //Do something and emit values ..................

   context.write(out, new Text(emit));
  }
 }
}

Happy Hadooping.......

18 comments:

  1. For latest and updated Cloudera certification dumps in PDF format contact us at completeexamcollection@gmail.com.
    Refer our blog for more details http://completeexamcollection.blogspot.in/2015/04/cloudera-hadoop-certification-dumps.html

    ReplyDelete



  2. That is very interesting; you are a very skilled blogger. I have shared your website in my social networks..!


    Online Reputation Management

    ReplyDelete
  3. Grateful to check out your website, I seem to be ahead to more excellent sites and I wish that you wrote more informative post for us. Well done work.

    SEO Company in Chennai

    ReplyDelete
  4. Its very useful to me. Wonderful blog.. Thanks for sharing informative Post.

    Installment loans
    Payday loans
    Title loans

    ReplyDelete
  5. it’s really nice and meanful. it’s really cool blog. Linking is very useful thing.you have really helped lots of people who visit blog and provide them usefull information.
    Hadoop Training in Hyderabad
    Java Training in Hyderabad

    ReplyDelete


  6. Its a wonderful post and very helpful, thanks for all this information. You are including better information regarding this topic in an effective way.Thank you so much

    Personal Installment Loans
    Payday Cash Advance loan
    Title Car loan
    Cash Advance Loan

    ReplyDelete
  7. Thanks for appreciating. Really means and inspires a lot to hear from you guys.I have bookmarked it and I am looking forward to reading new articles. Keep up the good work..Believe me, This is very helpful for me.

    Hadoop Training in Chennai

    Base SAS Training in Chennai

    MSBI Training in Chennai

    ReplyDelete
  8. Really useful post about hadoop, i have to some information about hadoop online training institute in india.

    ReplyDelete
  9. You have provided an nice article, Thank you very much for this one. And i hope this will be useful for many people.. and i am waiting for your next post keep on updating these kinds of knowledgeable things...
    Hadoop training in Gurgaon

    ReplyDelete
  10. Very amazing blog on a fresh new topic. I am definitely going to use the information provided in this blog because of its novel and fresh. Thanks for sharing such valuable information with everyone.
    Web Development Companies in Bangalore
    Website Design and Development Services in Bangalore
    ECommerce Web Design Company in India
    Outsource magento ecommerce services india

    ReplyDelete
  11. Nice blog. Thanks for sharing such great information.Inwizards Inc is a Hadoop Development services company offers quality Hadoop big data services best in web industries. Intrested click here - Hadoop development services in India

    ReplyDelete
  12. This is incredible posting! I quite enjoyed reading it, you happen to be a great author. I will remember to bookmark your blog and will eventually come back very soon. Also share with my community and friends about this.
    Web Development Services in Bangalore
    BEST web design services in bangalore
    ECommerce Website developers in bangalore
    Outsource magento ecommerce services india

    ReplyDelete
  13. A great content and very much useful to the visitors. Looking for more updates in future.

    BIGDATA Training in Chennai

    ReplyDelete
  14. hi ,
    your post on hadoop doublearraywritables was very useful in my codings do keep posting blogs... Hadoop Training in Velachery | Hadoop Training .
    Hadoop Training in Chennai | Hadoop .

    ReplyDelete
  15. AWS Training in Bangalore - Live Online & Classroom
    myTectra Amazon Web Services (AWS) certification training helps you to gain real time hands on experience on AWS. myTectra offers AWS training in Bangalore using classroom and AWS Online Training globally. AWS Training at myTectra delivered by the experienced professional who has atleast 4 years of relavent AWS experince and overall 8-15 years of IT experience. myTectra Offers AWS Training since 2013 and retained the positions of Top AWS Training Company in Bangalore and India.

    IOT Training in Bangalore - Live Online & Classroom
    IOT Training course observes iot as the platform for networking of different devices on the internet and their inter related communication. Reading data through the sensors and processing it with applications sitting in the cloud and thereafter passing the processed data to generate different kind of output is the motive of the complete curricula. Students are made to understand the type of input devices and communications among the devices in a wireless media.

    ReplyDelete
  16. Big Data and Hadoop training in Bangalore, offered by myTectra. India's No.1 Big Data and Hadoop training Institute. Classroom, Online and Corporate training.hadoop training in bangalore

    ReplyDelete
  17. mytectra placement Portal is a Web based portal brings Potentials Employers and myTectra Candidates on a common platform for placement assistance.

    ReplyDelete