@dksrinivasa i have one doubt , even if they ask us to run the script ? can we execute it in CLI one by one.
i mean i can run the script, but i am wondering if something goes wrong. can i execute it line by line in pyspark/scala shell as at the end the output will be the same (storing,aggregating etc etc).
i think , there would be two scripts , 1 main .sh script and a child script which will be .py/scala script. we need to fill the .py script and launch the .sh script (./script,sh)which will call the .py script and execute it