Data Engineering Spark SQL - Spark SQL Functions - Overview of Functions

In this article, we will explore the overview of predefined functions in Spark SQL. We will cover key concepts and hands-on tasks to help you understand and apply these functions effectively. Additionally, we will provide a video link that complements the text, offering a visual aid in understanding the concepts discussed.

[Embed video link here]

Key Concepts Explanation

Overview of Functions

Let us start by getting an overview of predefined functions in Spark SQL. We will demonstrate how to set up a Spark context in the Notebook for code execution.

val username = System.getProperty("")
import org.apache.spark.sql.SparkSession

val spark = SparkSession.
    config("spark.ui.port", "0").
    config("spark.sql.warehouse.dir", s"/user/${username}/warehouse").
    appName(s"${username} | Spark SQL - Predefined Functions").

List of Functions

You can use the following commands to interact with Spark SQL:

  • Using Spark SQL CLI:
spark2-sql \
    --master yarn \
    --conf spark.ui.port=0 \
    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse
  • Using Scala:
spark2-shell \
    --master yarn \
    --conf spark.ui.port=0 \
    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse
  • Using Pyspark:
pyspark2 \
    --master yarn \
    --conf spark.ui.port=0 \
    --conf spark.sql.warehouse.dir=/user/${USER}/warehouse

Categories of Functions

There are various categories of commonly used functions in Spark SQL, including:

  • String Manipulation
  • Date Manipulation
  • Numeric Functions
  • Type Conversion Functions
  • CASE and WHEN

You can explore more functions by running SHOW functions and use DESCRIBE FUNCTION to understand the syntax and semantics of a specific function like substr.

Hands-On Tasks

To practice the concepts discussed in this article, you can perform the following tasks:

  1. Run SHOW functions in Spark SQL to get a list of functions.
  2. Use DESCRIBE FUNCTION substr to understand the details of the substr function.


In conclusion, this article has provided an overview of Spark SQL predefined functions, explained key concepts, and offered hands-on tasks for practical application. We encourage you to practice these functions and engage with the community for further learning.

Remember, practice makes perfect!

Watch the video tutorial here