Dataframe error while running on window


#1

Hello-

I was running dataframe using window(10) based pyspark(Spark 2.3 and python 3) but getting some error with the path of the file in window. I need your help to resolve this issue.

I was going thru the below video.

My code

data = spark.read.csv(“E:\deckofcards.csv”)

error

Caused by: ERROR XBM0H: Directory C:\Windows\System32\metastore_db cannot be created.

Regards,
Jeevan

Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster


#2

This is resolved. You can close it.

Regards,
Jeevan


#3

Hi Jeevan,

Let us know How did you resolve this error in Windows.
I am also facing similar issue.


#4

Hi Surendra-

I followed the below steps.

Step 1:

Create C:\tmp\hive directory.

Note

c:\tmp\hive directory is the default value of hive.exec.scratchdir configuration property
in Hive 0.14.0 and later and Spark uses a custom build of Hive 1.2.1.

You can change hive.exec.scratchdir configuration property to another directory as
described in Changing hive.exec.scratchdir Configuration Property in this document.

Step 2:

Execute the following command in cmd that you started using the option Run as administrator.

winutils.exe chmod -R 777 C:\tmp\hive

Step 3:

Check the permissions (that is one of the commands that are executed under the covers):
winutils.exe ls -F C:\tmp\hive

Reference -
https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-tips-and-tricks-running-spark-windows.html

Warm Regards-
Jeevan