Object spark is not a member of package org.apache

apache-spark

#1

Hi,
I understand that there are multiple posts created for this issue.However I am not able to resolve the problem.

I am trying to create package using sbt in lab and getting the error.As suggested in previous topics Iverified the documentation and couldnt resove this error.
Error:
[chethanshekar@gw02 build]$ sbt package
[info] Loading project definition from /home/chethanshekar/build/project
[info] Compiling 1 Scala source to /home/chethanshekar/build/project/target/scala-2.10/sbt-0.13/classes…
[error] /home/chethanshekar/build/project/src/main/scala/StreamingDepCount.scala:1: object spark is not a member of package org.apache
[error] import org.apache.spark._
[error] ^
[error] /home/chethanshekar/build/project/src/main/scala/StreamingDepCount.scala:2: object spark is not a member of package org.apache
[error] import org.apache.spark.streaming._

Code:

[chethanshekar@gw02 scala]$ pwd
/home/chethanshekar/build/project/src/main/scala
[chethanshekar@gw02 scala]$ cat StreamingDepCount.scala
import org.apache.spark._
import org.apache.spark.streaming._
object StreamingDepCount {

def main(args:Array[String]) {
val conf = new SparkConf().
setAppName(“Streaming Department count”).
setMaster(args(0))
val ssc = new StreamingContext(conf,seconds(30))

val messages = ssc.socketTextStream(args(1),args(2).toInt)
val departmentMessges = messages.
filter(msg => {
val endpoint = msg.split(" “)(6)
endpoint.split(”/")(1) == “department”
})

val depratments = departmentMessages.
map(rec => {
val endpoint = msg.split(" “)(6)
(endpoint.split(”/")(2),1)
})

val departmentTraffic = departments.
reduceByKey((total,value) => total + value)
departmentTraffic.saveAsTextFiles(“output/problem1”)
ssc.start()
ssc.awaitTermination()
}
}

build.sbt
[chethanshekar@gw02 build]$ cat build.sbt
name:=“sparkt”
version:=“1.0”
libraryDependencies += “org.apache.spark” % “spark-core_2.11” % “2.3.1”
libraryDependencies += “org.apache.spark” % “spark-sql_2.11” % “2.3.1”


#2

@Chandra_Shekar Place build.sbt inside the location where the .scala file and ran sbt package


#3

Thanks it is working now


#4