How to install "spark-csv Databricks" package in Itversity

Hi Team,
Could someone guide me in installing com.databricks.spark.csv package so that i will be able to read csv file in a dataframe.

Thanks

Share the details about the associated jar file. We will add.

Also you can download to your home directory and pass the jar file using --jars

Its resolved now. I added maven dependency for Spark-csv into my eclipse and it worked.

1 Like

Switch to sbt from maven. It is not good practice to use maven for Scala based projects even though it is ok with respect to learning.

I am using below command to use databricks package:

spark-shell --jars spark-csv_2.10-1.4.0.jar

while usiing:

sqlContext.read.format(“com.databricks.spark.csv”).load("/user/sparklearner/person")

it is giving below error:

java.lang.NoClassDefFoundError: org/apache/commons/csv/CSVFormat
at com.databricks.spark.csv.package$.(package.scala:27)
at com.databricks.spark.csv.package$.(package.scala)
at com.databricks.spark.csv.CsvRelation.inferSchema(CsvRelation.scala:235)
at com.databricks.spark.csv.CsvRelation.(CsvRelation.scala:73)
at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:162)
at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:44)
at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:158)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:119)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:109)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:26)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:31)
at $iwC$$iwC$$iwC$$iwC$$iwC$$iwC.(:33)
at $iwC$$iwC$$iwC$$iwC$$iwC.(:35)
at $iwC$$iwC$$iwC$$iwC.(:37)
at $iwC$$iwC$$iwC.(:39)
at $iwC$$iwC.(:41)

@itversity can u plz guide me on this?

thanks in advance.

Hi,
Add com.databricks:spark-csv package to your spark-submit job. it should work…

It will look like below

spark-submit --packages com.databricks:spark-csv_2.11:1.3.0 --class com.db.SparkSql.CSVReader --master yarn-client SparkSql-0.0.1-SNAPSHOT.jar

1 Like