Apache Spark Python - Processing Column Data - Date and Time Manipulation Functions

Let us get started with Date and Time manipulation functions. As part of this topic, we will focus on the date and timestamp format.

  • We can use current_date to get today’s server date.

    • Date will be returned using yyyy-MM-dd format.
  • We can use current_timestamp to get the current server time.

    • Timestamp will be returned using yyyy-MM-dd HH:mm:ss:SSS format.
    • Hours will be by default in 24-hour format.

current_date()

The current_date() function returns the current server date in the format yyyy-MM-dd.

df.select(current_date()).show()

current_timestamp()

The current_timestamp() function returns the current server timestamp in the format yyyy-MM-dd HH:mm:ss.SSS.

df.select(current_timestamp()).show(truncate=False)

to_date() and to_timestamp()

We can convert a string that contains a date or timestamp in a non-standard format to a standard date or time using to_date or to_timestamp function, respectively.

df.select(to_date(lit('20210228'), 'yyyyMMdd').alias('to_date')).show()
df.select(to_timestamp(lit('20210228 1725'), 'yyyyMMdd HHmm').alias('to_timestamp')).show()

Watch the video tutorial here

Hands-On Tasks

Description of the hands-on tasks. Provide a list of tasks that the reader can perform to apply the concepts discussed in the article.

  1. Execute the current_date() function and observe the output.
  2. Use the to_date() function to convert a non-standard date format string to a standard date format.

Conclusion

In this article, we covered important date and time manipulation functions in Spark SQL. By understanding and applying these functions, you can effectively work with date and timestamp data in Spark. Practice these concepts in your Spark environment and feel free to engage with the community for further learning.