Spark.read.csv failing with permission issue


#1

Hi Team,

I am trying to load and create a data frame in spark 2.But i am getting an error message which is given below.It looks like some permission issue.Can u please give me a guidelines about this issue.

val data =spark.read.csv("/user/syeswanth/Files/sample.csv")
18/04/21 08:55:29 WARN RetryInvocationHandler: Exception while invoking ClientNamenodeProtocolTranslatorPB.getFileInfo over null. Not retrying
because try once and fail.
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.AccessControlException): Permission denied: user=syeswanth, access=EXECUTE, in
ode="/user/syeswanth/Files/sample.csv/_spark_metadata":syeswanth:hdfs:-rw-r–r--
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:319)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkTraverse(FSPermissionChecker.java:259)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:205)
at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1827)
at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getFileInfo(FSDirStatAndListingOp.java:108)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3972)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:1130)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslato
rPB.java:851)


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster


#2

We need to pass directory not file path to spark.read.csv.

Here is the working example.

scala> val data =spark.read.csv("/user/syeswanth/Files")
data: org.apache.spark.sql.DataFrame = [_c0: string, _c1: string … 5 more fields]