Thursday, 8 November 2018

How to do an aggregate function on a Spark Dataframe using collect_set

›
In order to explain usage of collect_set, Lets create a Dataframe with 3 columns. spark-shell --queue= *; To adjust logging leve...
3 comments:
Friday, 26 October 2018

How to add multiple withColumn to Spark Dataframe

›
In order to explain, Lets create a dataframe with 3 columns spark-shell --queue= *; To adjust logging level use sc.setLogL...
7 comments:
Saturday, 3 March 2018

Conditional Join in Spark using Dataframe

›
Lets see how can we add conditions along with dataframe join in spark Say we have 2 dataframes: dataFrame1,dataFrame2 val dataFrame1 = ...
1 comment:
Thursday, 10 December 2015

Faster way to count number of lines in a file/dir using Map Reduce Framework

›
In this site you can see one way to count number of lines in a file. They are emitting count as one for each record in each map. So if...
15 comments:

DoubleArrayWritable in Hadoop

›
Lets see how to emit double arrays from mapper and process them in reducer DoubleArrayWritable class public static class DoubleArray...
11 comments:
Wednesday, 9 December 2015

Partitioning Data Using Hadoop MultipleOutputs

›
There may be cases where we need to partition our data based on certion condition. Say for example, Consider this Employee data EmpId,...
16 comments:
Monday, 4 May 2015

Hadoop Word Count Using C Language - Hadoop Streaming

›
Prerequisites 1. Hadoop (Example based on cloudera distribution cdh5) 2. gcc compiler Hadoop streaming is a utility that comes wit...
17 comments: