I have an urgent concern regarding the examination (attempting with python) which is due for me in next 2 days.
Question- Importing avro packages in pyspark during the examination?
I have been getting mixed answers from people, some say we have to load it manually and some say it is pre-loaded in the environment.
I have been using following script all this while on itversity labs-
pyspark --master yarn –packages com.databricks:spark-avro_2.10:2.0.1 --conf spark.ui.port=xxxxxx
But unfortunately, few of the folks couldn’t make it work during the examination and wasted their attempt & money. I’m really getting nervous as many of the people recently have received multiple questions around avro data packages. I will really appreciate an early reply from anyone who has an absolute solution to this problem.
Thanks in advance!