Let us see an example of how we can read data from a file into list of dicts using Python as the programming language.
When we read data from a file into a list, typically each element in the list will be of type binary or string.
We can convert each element into a dict to simplify the processing.
Once each element is converted to a dict, we can access elements in the dict using attribute name.
Let’s dive into an example to read the data from a file into a list of dicts and access dates.
Reading data from file into a list:
path = '/data/retail_db/orders/part-00000'
# C:\\users\\itversity\\Research\\data\\retail_db\\orders\\part-00000
orders_file = open(path)
orders_raw = orders_file.read()
orders = orders_raw.splitlines()
orders[:10]
len(orders) # same as the number of records in the file
def get_order_dict(order):
order_details = order.split(',')
order_dict = {
'order_id': int(order_details[0]),
'order_date': order_details[1],
'order_customer_id': int(order_details[2]),
'order_status': order_details[3],
}
return order_dict
get_order_dict(orders[0])
order_dicts = [get_order_dict(order) for order in orders]
type(order_dicts)
type(order_dicts[0])
order_dicts[0]
order_dicts[:3]
len(order_dicts)
order_dates = [order['order_date'] for order in order_dicts]
order_dates[:3]
len(order_dates)
set(order_dates)
order_customer_ids = [order['order_customer_id'] for order in order_dicts]
order_customer_ids[:3]
type(order_customer_ids[0])
# Reading data from file into a list
path = '/data/retail_db/orders/part-00000'
# C:\\users\\itversity\\Research\\data\\retail_db\\orders\\part-00000
orders_file = open(path)
orders_raw = orders_file.read()
orders = orders_raw.splitlines()
order_dicts = [get_order_dict(order) for order in orders]
order_dates = [order['order_date'] for order in order_dicts]
order_dates[:3]
len(order_dates)