Saturday, 3 March 2018

Conditional Join in Spark using Dataframe

Lets see how can we add conditions along with dataframe join in spark
Say we have 2 dataframes: dataFrame1,dataFrame2
val dataFrame1 = hc.sql("select * from tbl1") //id,name,code
val dataFrame2 = hc.sql("select * from tbl2") //id,name,code

We need to join these 2 df's with different columns based on condition.

We have a decision Flag coming in with true/false value.
If the decision flag is true we need to set join condition with id and code columns else only with id column.

So how can we achieve this in scala.
val decision: Boolean = false

Lets set an expression
val exprs = (if (decision != true) 
             else dataFrame1.col("id").equalTo(dataFrame2.col("id")) 
                   and dataFrame1.col("code").equalTo(dataFrame2.col("code")))

and then join
dataFrame1.join(dataFrame2, exprs).show

This is how you join 2 dataframes with conditions.


  1. Its really an Excellent post. I just stumbled upon your blog and wanted to say that I have really enjoyed reading your blog. Thanks for sharing....

    Restaurant in OMR
    Apartments in OMR
    Villas in OMR
    Resorts in OMR

  2. Thanks for the informative article. This is one of the best resources I have found in quite some time.Nicely written and great info. I really cannot thank you enough for sharing.

    Herbalife in chennai
    wellnesscoaches in chennai
    Weightloss in chennai
    Weightgain in chennai

  3. All the latest updates from the Python Automationminds team. Python Automationminds lets you program in Python, in your browser. No need to install any software, just start coding straight away. There's a fully-functional web-based console and a programmer's text-editor
    Phyton training in Chennai

  4. Testers can build, enhance, and maintain scripts to regression test their mobile applications. Hands-on instruction is provided for those who want to explore the power of using Appium. The course covers content from installation to execution and reporting . The focus is on the practical application of Appium to resolve common mobile automated testing challenges. This course focuses on getting started with Appium.

  5. Nice post. By reading your blog, i get inspired and this provides some useful information. Thank you for posting this exclusive post for our vision.
    Selenium Training in Chennai | Selenium Training in Bangalore | Selenium Training in Pune | Selenium online Training

  6. myTectra Placement Portal is a Web based portal brings Potentials Employers and myTectra Candidates on a common platform for placement assistance

  7. SQream Technologies provides you with a state of the art software which combines modern GPU technology (Graphic Processing Units) with the best practices in today’s Big Data platforms, providing up to 100x faster insights from data.
    Bigdata Training in Chennai OMR

  8. Thanks for providing such a great Information, you can see
    Peptide Synthesis