Data Engineering Spark SQL - Managing Tables - DDL & DML - Loading Data - Append and Overwrite

In this article, we will explore different approaches to load data into a Spark Metastore table. We will cover how to append data into an existing table and how to overwrite data in a table.

Explanation for the video

[Embed Video Here]

Key Concepts Explanation

Append Data into Table

To append data into an existing table, we use the INTO TABLE clause in Spark SQL. This will add the new data to the existing data in the table.

LOAD DATA LOCAL INPATH '/data/retail_db/orders' 

Overwrite Data in Table

To overwrite data in a table, we specify OVERWRITE INTO TABLE in Spark SQL. This will replace the existing data in the table with the new data.

LOAD DATA LOCAL INPATH '/data/retail_db/orders' 

Hands-On Tasks

  1. Append data into the ‘orders’ table using the provided data file.
  2. Overwrite data in the ‘orders’ table using the same data file.


In this article, we discussed how to load data into Spark Metastore tables by either appending data to an existing table or overwriting data in a table. It is essential to understand these concepts to manage and update data effectively in Spark. Practice these tasks and engage with the community for further learning.

Watch the video tutorial here