Understanding Spark 2.0 Catalog API

Catalog API

DataSet with Dataframe API supports structured data analysis in spark. One of the important aspects of structured data analysis is managing metadata. It may be temporary metadata like temp table, registered udfs on SQL context or permanent metadata like Hive meta store or HCatalog.

In earlier versions of spark, there was no standard API to access this metadata. Users used to use queries like show tables and others to query this metadata. These queries often needed raw string manipulation and used to differ depending upon the underneath meta store.

But it’s changing in Spark 2.0.In Spark 2.0, spark has added a standard API called catalog for accessing metadata in spark SQL. This works both for spark sql and hive metadata.

You can access complete code :