Wednesday, 2 January 2019

How to add Scalastyle plugin in eclipse

›
Scalastyle is a style checker for Scala. It checks your Scala code against a number of configurable rules, and marks the code which viola...
19 comments:
Thursday, 13 December 2018

error: not found: type Column in Spark Scala

›
How to get rid of below error spark-shell --queue= *; To adjust logging level use sc.setLogLevel(newLevel). Welcome to ____ ...
5 comments:
Thursday, 8 November 2018

How to do an aggregate function on a Spark Dataframe using collect_set

›
In order to explain usage of collect_set, Lets create a Dataframe with 3 columns. spark-shell --queue= *; To adjust logging leve...
3 comments:
Friday, 26 October 2018

How to add multiple withColumn to Spark Dataframe

›
In order to explain, Lets create a dataframe with 3 columns spark-shell --queue= *; To adjust logging level use sc.setLogL...
7 comments:
Saturday, 3 March 2018

Conditional Join in Spark using Dataframe

›
Lets see how can we add conditions along with dataframe join in spark Say we have 2 dataframes: dataFrame1,dataFrame2 val dataFrame1 = ...
1 comment:
Thursday, 10 December 2015

Faster way to count number of lines in a file/dir using Map Reduce Framework

›
In this site you can see one way to count number of lines in a file. They are emitting count as one for each record in each map. So if...
15 comments:

DoubleArrayWritable in Hadoop

›
Lets see how to emit double arrays from mapper and process them in reducer DoubleArrayWritable class public static class DoubleArray...
11 comments: