Issue in Creating Project Using Spark 2.4.5 in Intellij

Hello,
I am new to Spark. When i try creating a new project, sbt task fails.
This is my intellij screen.


sbt version is 0.13.17
my doubts are which scala version and sbt version should i use? and when i see the official documentation it shows dependancies with scala 2.12?
Thank you.

1 Like

When deciding on dependencies, always try and compare what you have with the maven dependency references.

E.g. I am looking for an SBT dependency for Spark streamin 2.4.5 compiled with Scala 2.12.
I go to the MVN dependency page for Spark streaming here: Maven spark streaming

I then select the tab for SBT:
image

As this shows the correct term is:
// https://mvnrepository.com/artifact/org.apache.spark/spark-streaming
libraryDependencies += “org.apache.spark” %% “spark-streaming” % “2.4.5” % “provided”

As you are probably running this on a cluster they use the provided scope as this marks **dependencies that should be provided at runtime by JDK or a container **.