Apache Spark Python - Processing Column Data - Date and Time Extract Functions

Let us get an overview about Date and Time extract functions. Here are the extract functions that are useful which are self explanatory.

  • year

  • month

  • weekofyear

  • dayofyear

  • dayofmonth

  • dayofweek

  • hour

  • minute

  • second

There might be few more functions. You can review based up on your requirements.
Let us start spark context for this Notebook so that we can execute the code provided. You can sign up for our 10 node state of the art cluster/labs to learn Spark SQL using our unique integrated LMS.

Date Extraction Functions

The year, month, weekofyear, dayofyear, dayofmonth, dayofweek, hour, minute, and second functions in Spark SQL are used to extract specific components of a date or timestamp. Here is an example of how to use these functions:

from pyspark.sql.functions import year, month, weekofyear, dayofmonth, dayofyear, dayofweek, current_date

df.select(
    current_date().alias('current_date'), 
    year(current_date()).alias('year'),
    month(current_date()).alias('month'),
    weekofyear(current_date()).alias('weekofyear'),
    dayofyear(current_date()).alias('dayofyear'),
    dayofmonth(current_date()).alias('dayofmonth'),
    dayofweek(current_date()).alias('dayofweek')
).show()

Time Extraction Functions

The current_timestamp, hour, minute, and second functions in Spark SQL are used to extract specific components of a timestamp. Here is an example of how to use these functions:

from pyspark.sql.functions import current_timestamp, hour, minute, second

df.select(
    current_timestamp().alias('current_timestamp'), 
    year(current_timestamp()).alias('year'),
    month(current_timestamp()).alias('month'),
    dayofmonth(current_timestamp()).alias('dayofmonth'),
    hour(current_timestamp()).alias('hour'),
    minute(current_timestamp()).alias('minute'),
    second(current_timestamp()).alias('second')
).show(truncate=False)

Explore these date and time extract functions further with our video tutorial! Watch as we demonstrate how to use these functions to extract specific components of date and timestamp data in Spark SQL. Enhance your understanding and learn practical applications of these functions by watching our tutorial now

Watch the video tutorial here

Hands-On Tasks

Description of the hands-on tasks. Provide a list of tasks that the reader can perform to apply the concepts discussed in the article.

  1. Perform date and time extraction tasks using the provided functions.
  2. Explore additional date and time extraction functions based on your requirements.

Conclusion

In this article, we explored the date and time extract functions available in Spark SQL. By using these functions, you can easily extract specific components of date and timestamp values. We encourage you to practice these concepts and engage with the community for further learning.