Thursday, 24 September 2020

Create Delta table on csv file in python spark

 

You can read files into Dataframe and write out in delta format

Step 1 : Read the input csv

Step 2 : Write the csv to ADLS location using Delta format

Step 3: Create a table on top of it


myCSV= spark.read.csv("/path/to/input/data",header=True,sep=","); 
myCSV.write.format("delta").mode("overwrite").option('overwriteSchema','true').save("/mnt/delta/Employee") 
spark.sql("CREATE TABLE employee USING DELTA LOCATION '/mnt/delta/Employee/'") 

1 comment:

  1. Learn AWS is the best way to get more salary in Industry. Cognex offers so many courses like aws, microsoft azure, prince 2 foundation, etc,. cognex is well known for aws training

    ReplyDelete