How to create Dataframe from a raw file xxxx.dat which is in compressed format. I have json schema separately which have startIndex, endIndex, fieldName and fieldType. Now the question is how to match the json schema with xxxx.dat file to produce Dataframe…? Then i have to convert the Dataframe into parquet file with .gz compression.
Anybody have an idea about this usecase…?