Efficient Data Processing on Kubernetes with PySpark, Pandas & Polars Training Course

Module 1: Big Data & Spark Foundations

• Overview of the Big Data landscape and Spark's pivotal role in contemporary data platforms
• Exploring Spark architecture: drivers, executors, cluster managers, lazy evaluation, DAGs and execution planning
• Distinguishing between RDD and DataFrame APIs and determining the optimal approach for specific use cases
• Establishing and configuring SparkSession, along with understanding the fundamentals of application configuration

Module 2: PySpark DataFrames

• Ingesting and exporting data from enterprise sources and formats, including CSV, JSON, Parquet and Delta
• Utilising PySpark DataFrames for transformations, actions, column expressions, filtering, joins and aggregations
• Implementing advanced operations such as window functions, timestamp handling and managing nested data structures
• Integrating data quality checks and developing reusable, maintainable PySpark code

Module 3: Kubernetes & Docker Fundamentals for Spark Users

• Comprehending the relationship between Docker images, Kubernetes pods and Spark driver/executor processes, from a cluster consumer perspective rather than an administrative one
• Understanding the mechanics of Spark jobs on Kubernetes: pod scheduling, resource requests/limits, and the mapping of session settings to cluster resources
• Distinguishing between controls managed via Spark session parameters versus those handled by platform teams during cluster provisioning, and why this boundary is critical for tuning
• Practical exercise: Examine an active Spark-on-Kubernetes job in the sandbox cluster to identify driver and executor pods

Module 4: Spark Session Configuration on Kubernetes — Deep Dive

• Analyzing the trade-offs between executor count and executor size, focusing on memory-versus-cores considerations
• Allocating driver and executor memory, managing memory overhead, and translating these settings into concrete pod resources
• Evaluating dynamic allocation: its behaviour on Kubernetes, scenarios where it conserves resources and those where it may not
• Practical exercise: Execute identical jobs under varying executor and core configurations to analyse differences in runtime and resource consumption

Module 5: Scaling Behaviour & Cost Optimisation

• Assessing how scaling nodes up or down affects job behaviour, completion times and overall resource consumption
• Comparing strategies of using many small executors versus fewer large ones, weighing performance against cost
• Managing shuffle behaviour and tuning shuffle partitions, including estimating the cost implications of different configuration choices
• Practical exercise: Scale the sandbox cluster from five to ten nodes to observe the impact on job completion time and resource usage

Module 6: Efficient Data Ingestion & Partitioning

• Addressing the small-files problem: understanding why splitting sources into numerous 1–5 MB Parquet files degrades performance and distorts partitioning
• Implementing repartitioning and coalescing strategies for optimal data flow
• Controlling partition sizes during both read and write operations
• Writing Parquet data efficiently to prevent the recurrence of small-files issues in downstream processes
• Practical exercise: Load a dataset composed of many small Parquet partitions, test various repartitioning strategies and evaluate performance improvements

Module 7: Pandas Memory Management & Failure Diagnosis

• Identifying root causes of out-of-memory errors in Pandas and recognising key symptoms
• Applying memory-efficient patterns for converting data between Spark and Pandas
• Preventing memory spikes when writing large datasets to CSV
• Utilising chunked processing and dtype optimisation for environments with constrained memory
• Practical exercise: Recreate a common Pandas out-of-memory scenario and resolve it using chunking and dtype optimisation techniques

Module 8: Polars as a Complementary Tool

• Positioning Polars in relation to Pandas: examining performance characteristics, lazy evaluation and memory behaviour
• Understanding the role of Polars alongside PySpark and Pandas in modern cloud data stacks and migration roadmaps, particularly in AWS environments
• Practical exercise: Rewrite a Pandas-heavy transformation in Polars and compare memory usage and processing speeds

Module 9: Applying Optimisation to ETL & ML Workloads

• Applying configuration, partitioning and memory-management principles across a realistic ETL pipeline
• Considering optimisation requirements specific to machine-learning workloads running on the same cluster
• Implementing a structured tuning workflow to systematically diagnose cost and performance issues
• Practical exercise: Complete an end-to-end mini-project involving data loading, transformation and simple model training, with participants independently tuning the Spark configuration