Xml Data processing in spark - Process finished with exit code 1 in intellij


#1

Hi ,
This is my code

This is my .sbt
image

Using Spark’s default log4j profile: org/apache/spark/log4j-defaults.properties
18/08/21 17:48:18 INFO SparkContext: Running Spark version 2.2.0
18/08/21 17:48:18 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform… using builtin-java classes where applicable
18/08/21 17:48:19 INFO SparkContext: Submitted application: ProcessXML
18/08/21 17:48:19 INFO SecurityManager: Changing view acls to: rajkiranu
18/08/21 17:48:19 INFO SecurityManager: Changing modify acls to: rajkiranu
18/08/21 17:48:19 INFO SecurityManager: Changing view acls groups to:
18/08/21 17:48:19 INFO SecurityManager: Changing modify acls groups to:
18/08/21 17:48:19 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rajkiranu); groups with view permissions: Set(); users with modify permissions: Set(rajkiranu); groups with modify permissions: Set()
18/08/21 17:48:20 INFO Utils: Successfully started service ‘sparkDriver’ on port 57522.
18/08/21 17:48:20 INFO SparkEnv: Registering MapOutputTracker
18/08/21 17:48:20 INFO SparkEnv: Registering BlockManagerMaster
18/08/21 17:48:20 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
18/08/21 17:48:20 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
18/08/21 17:48:20 INFO DiskBlockManager: Created local directory at C:\Users\rajkiranu\AppData\Local\Temp\blockmgr-6ecb3a19-5a12-4f86-a79c-a58313f7179a
18/08/21 17:48:20 INFO MemoryStore: MemoryStore started with capacity 1442.4 MB
18/08/21 17:48:20 INFO SparkEnv: Registering OutputCommitCoordinator
18/08/21 17:48:20 INFO Utils: Successfully started service ‘SparkUI’ on port 4040.
18/08/21 17:48:20 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.56.1:4040
18/08/21 17:48:20 INFO Executor: Starting executor ID driver on host localhost
18/08/21 17:48:21 INFO Utils: Successfully started service ‘org.apache.spark.network.netty.NettyBlockTransferService’ on port 57532.
18/08/21 17:48:21 INFO NettyBlockTransferService: Server created on 192.168.56.1:57532
18/08/21 17:48:21 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
18/08/21 17:48:21 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.56.1, 57532, None)
18/08/21 17:48:21 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.56.1:57532 with 1442.4 MB RAM, BlockManagerId(driver, 192.168.56.1, 57532, None)
18/08/21 17:48:21 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.56.1, 57532, None)
18/08/21 17:48:21 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.56.1, 57532, None)
18/08/21 17:48:21 INFO SharedState: Setting hive.metastore.warehouse.dir (‘null’) to the value of spark.sql.warehouse.dir (‘file:/C:/Users/rajkiranu/IdeaProjects/DemoScala/spark-warehouse/’).
18/08/21 17:48:21 INFO SharedState: Warehouse path is ‘file:/C:/Users/rajkiranu/IdeaProjects/DemoScala/spark-warehouse/’.
18/08/21 17:48:22 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
Exception in thread “main” java.lang.NoSuchMethodError: org.apache.spark.sql.types.DecimalType$.Unlimited()Lorg/apache/spark/sql/types/DecimalType;
at com.databricks.spark.xml.util.InferSchema$.(InferSchema.scala:36)
at com.databricks.spark.xml.util.InferSchema$.(InferSchema.scala)
at com.databricks.spark.xml.XmlRelation$$anonfun$1.apply(XmlRelation.scala:52)
at com.databricks.spark.xml.XmlRelation$$anonfun$1.apply(XmlRelation.scala:52)
at scala.Option.getOrElse(Option.scala:121)
at com.databricks.spark.xml.XmlRelation.(XmlRelation.scala:51)
at com.databricks.spark.xml.DefaultSource.createRelation(DefaultSource.scala:89)
at com.databricks.spark.xml.DefaultSource.createRelation(DefaultSource.scala:43)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:306)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:178)
at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:156)
at Dataframes.DataFrame2$.xmlprocess(DataFrame2.scala:27)
at Dataframes.DataFrame2$.main(DataFrame2.scala:18)
at Dataframes.DataFrame2.main(DataFrame2.scala)
18/08/21 17:48:22 INFO SparkContext: Invoking stop() from shutdown hook
18/08/21 17:48:22 INFO SparkUI: Stopped Spark web UI at http://192.168.56.1:4040
18/08/21 17:48:22 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
18/08/21 17:48:22 INFO MemoryStore: MemoryStore cleared
18/08/21 17:48:22 INFO BlockManager: BlockManager stopped
18/08/21 17:48:22 INFO BlockManagerMaster: BlockManagerMaster stopped
18/08/21 17:48:22 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
18/08/21 17:48:22 INFO SparkContext: Successfully stopped SparkContext
18/08/21 17:48:22 INFO ShutdownHookManager: Shutdown hook called
18/08/21 17:48:22 INFO ShutdownHookManager: Deleting directory C:\Users\rajkiranu\AppData\Local\Temp\spark-7ccef0ef-8f69-426a-9d26-a657fdf30dc4


Learn Spark 1.6.x or Spark 2.x on our state of the art big data labs

  • Click here for access to state of the art 13 node Hadoop and Spark Cluster