Apache Spark Python - Transformations - Solution - Get Flight Count Per US State

Data Engineering Spark
1

This article provides a step-by-step guide on analyzing the number of flight departures from each US state in January 2008 using Apache Spark. The article covers concepts like joining datasets, filtering data, and aggregating results to gain insights into air traffic patterns.

Spark Context Initialization

To start analyzing air traffic data using Apache Spark, we need to initialize the Spark session and configure necessary settings to support our analysis. The code snippet demonstrates how to set up a Spark session for processing the air traffic dataset.

from pyspark.sql import SparkSession

spark = SparkSession. \
    builder. \
    config('spark.ui.port', '0'). \
    config("spark.sql.warehouse.dir", f"/user/{username}/warehouse"). \
    enableHiveSupport(). \
    appName(f'{username} | Python - Joining Data Sets'). \
    master('yarn'). \
    getOrCreate()

Joining and Aggregating Data

To analyze the number of flight departures from each US state, we need to join the air traffic data with airport codes based on the Origin airport code. This code snippet demonstrates how to perform an inner join between the air traffic and airport codes datasets to calculate the flight count for each state.

from pyspark.sql.functions import col, lit, count

airtraffic. \
    join(airportCodes, col("IATA") == col("Origin"), "inner"). \
    groupBy("State"). \
    agg(count(lit(1)).alias("FlightCount")). \
    orderBy(col("FlightCount").desc()). \
    show()

Watch the video tutorial here

Hands-On Tasks

  1. Initialize the Spark session to analyze air traffic data.
  2. Perform an inner join between the air traffic and airport codes datasets.
  3. Group the data by state and calculate the number of flight departures.

Conclusion

In this article, we explored how to analyze the number of flight departures from each US state in January 2008 using Apache Spark. By following the provided instructions and code examples, readers can gain valuable insights into air traffic patterns and practice working with Spark SQL for data analysis.

Solutions - Problem 2

Get the number of flights departed from each US state in January 2008. Use Spark to join air traffic data and airport codes to aggregate flight counts by state.