Unable to avro data in spark shell


#1

Hi ,

i am unable to load avro data in

val avroData=sqlContext.read.avro("/user/ethabhi2795/problem5/avro")

-A


#2

Have you launch spark shell with below package

–packages com.databricks:spark-avro_2.10:2.0.1

And import com.databricks.spark.avro._. ???


#3

I used –packages com.databricks:spark-avro_2.10:2.0.


#4

I have imported

Import com.databricks.spark.avro._


#5

@Vani_Cs

Can you please help i am stil not able to import avro data


#6

Import com.databricks.spark.avro._


#7

I imported Import com.databricks.spark.avro._

But i am getting error object databricks is not a member of package


#8

@mohan or @BaLu_SaI Can you look into this issue pls
@paslechoix this user is also getting same error as like me
I imported Import com.databricks.spark.avro._ in labs

But i am getting error object databricks is not a member of package


#9

@ethabhi2795

As labs are in Hortonworks distribution, For Avro files, you have to invoke spark shell with avro package which will be as below:

spark-shell --packages com.databricks:spark-avro_2.10:2.0.1 --master yarn --conf spark.ui.port=22222

After this:

import com.databricks.spark.avro._

Now run DFname.write.avro(“ouput_path”)

But In exam as it is on Cloudera distribution, You only have to use:

import com.databricks.spark.avro._


#10

Thanks for the info @Raman_Kumar_Jha it is working fine