Not able to write DataFrame output to a parquet file

Not able to write DataFrame output to a parquet file into HDFS. It’s throwing permission issue while writting.

Hi @suresh_meher,
please go through below code snippet for your reference-

after launching pyspark2 (pyspark2 --master yarn --conf spark.ui.port=0) -
try-

creating dataframe-

ordersCSV = spark.read.csv('/public/retail_db/orders'). \
  toDF('order_id', 'order_date', 'order_customer_id', 'order_status')

changing data type of requared coloumn(by default spark DF takes everything as string type)

from pyspark.sql.types import IntegerType, FloatType
orders = ordersCSV. \
  withColumn('order_id', ordersCSV.order_id.cast(IntegerType())). \
  withColumn('order_customer_id', ordersCSV.order_customer_id.cast(IntegerType()))

writing dataframe into parquet file format-

orders.write.parquet('/user/username/.../')

still facing issue, please provide us your code which you tried.

Thanks for your patience