Background
I am a Senior Data Engineer at CloudMesh360. I mentored Simona, a bright intern who joined Northwind Traders last week.
Simona is an Excel Expert. She is used to:
1. Opening a file and seeing data immediately.
2. Saving a file and getting one single .xlsx or .csv file.
She just ran her first PySpark notebook on the 375 MB Sales Data. She believes the system is broken
because it is behaving "illogically."
The Challenge (Simona's Confusion)
Simona sent me an urgent message:
"Nomaan, I think the cluster is broken.
1. It's lying about speed: I loaded the 375 MB file, and it finished in 0.1 seconds. That is physically
impossible. I don't think it actually read the data.
2. It destroyed my file: I tried to save the report to the Volume. I expected one file named Report.csv.
Instead, I got a folder with 6 or 8 weird file fragments inside. I can't send this to the CEO!"
Task
I ran Simona's code (provided below) to replicate her findings. Then, I answered her some questions using my knowledge of Distributed Computing.
Workspace
Sample Codes on Databricks Notebook
Actionables
Importing all the functions and datatypes of pyspark library
Cell 1
Below line of code uses the Spark session to read a CSV file located at the specified path within a Unity Catalog Volume. By setting the header option to true, it ensures the first row of the file is treated as column names rather than data, storing the resulting structure into a DataFrame named df.
Cell 2
Below line of code exports the contents of the DataFrame df to the specified directory path in a Unity Catalog Volume using the CSV format. The .mode("overwrite") configuration ensures that any existing data at that target location is deleted and replaced with the new output.
Cell 3
String Manipulation
Creating a column Region_Code taking the first 3 characters of Region.
Creating a column Transaction_Key combining Region and TransactionID with a hyphen.
Type Casting & Handling
Converting the Date column from StringType to DateType.
Converting the Date column to TimestampType
Complex Logic & Filtering
Filtering where Region is "North" AND Sales > 100. (Multi Condition filter)
Creating a column Is_Bulk that is True if Quantity > 10.
Creating a column Net_Sales (Sales / 1.05) assuming sales included 5% tax.
Sorting the dataframe by Date (Oldest to Newest)
Sorting the dataframe by Sales (Highest to Lowest).
Sorting first by Region (A-Z), then by Sales (High-Low).
