Method Not Found exception :org.apache.spark.sql.DataFrameReader.load


I am trying to execute a spark-submit command mentioned below.But end up getting an error.


[vijaylv89@gw01 ~]$ spark-submit --packages com.databricks:spark-avro_2.10:2.0.1 --class --executor-memory 3712m --executor-cores 4–spark.ui.port=4044 sparkjar.jar


" Exception in thread “main” java.lang.NoSuchMethodError: org.apache.spark.sql.DataFrameReader.load(Ljava/lang/String;)Lorg/apache/spark/sql/Dataset;"
I am doing some repartition operations on a DF with Json data. Could not include correct package for the method “DataFrameReader.load”.

I need the right package to include.

Any leads would be appreciated. Thanks

@VIJAY_K It would be really helpful to debug the issue, if you can paste the code here.

Thanks for replying me.

Below is the code.


import org.apache.spark.SparkConf
import org.apache.spark.SparkContext
import org.apache.spark.sql.SQLContext

object repartition extends App {

val conf = new SparkConf().setAppName(“Assets_Repartitioned”).setMaster(“yarn-client”)

val sc = new SparkContext(conf);

val sqlContext = new SQLContext(sc)

val asseDF =“json”).load(“assets_final”)

val repart_assets = asseDF.repartition(8)

val repart_exe =“pv”, “rf”, “pid”).filter(“pid = 10266”)“Assets_RepartitionedO”)


I am able to run this code successfully in Scala IDE but not using spark-submit