What Is PySpark? Python API for Apache Spark

PySpark lets you write distributed data processing code in Python. Under the hood, your DataFrame operations compile to the same execution plan as Scala. The Catalyst optimizer treats them identically, applying 100+ optimization rules before generating a physical plan.

The DataFrame API was introduced in Spark 1.3 (2015) and replaced RDDs as the primary interface. Today, Spark 3.5 ships with 1,500+ built-in functions in pyspark.sql.functions. Interviewers expect you to use these functions instead of writing Python UDFs, which serialize data between the JVM and Python for every row.

Every major data platform runs Spark: Databricks, AWS EMR, Google Dataproc, Azure Synapse. A typical production cluster runs 50 to 500 executors with 4 to 8 cores each. If a company processes more than a few terabytes daily, they are almost certainly running Spark.

Knowing PySpark is not optional for data engineering roles at these companies. Interviewers test whether you have built and maintained a real project, not whether you can recite documentation. The tradeoff questions matter: when to broadcast vs. sort-merge join, when to cache vs. recompute, when to repartition vs. coalesce.

Pandas runs on a single machine and holds everything in memory. PySpark distributes work across a cluster. For datasets under 10GB, Pandas is simpler and faster to iterate with. For datasets over 10GB, PySpark is necessary.

The key behavioral difference: PySpark operations are lazy. Nothing executes until you call an action like .show(), .count(), or .write(). Pandas operations execute immediately. This catches people who call .count() inside a loop, forcing a full DAG evaluation on every iteration.

Both compile to the same physical plan. Performance is identical for DataFrame operations. The difference appears with UDFs: Python UDFs serialize data between the JVM and Python, adding overhead that can make them 10x to 100x slower than native functions. Pandas UDFs (vectorized) close most of this gap by operating on Arrow batches instead of individual rows.

PySpark accounts for roughly 70% of Spark API usage based on GitHub activity, Scala about 25%, Java about 5%. Choose PySpark if your team writes Python. Choose Scala if you need custom RDD operations or maximum UDF performance. For interview prep, the concepts are identical across both languages.

When you write df.filter(...).groupBy(...).agg(...), nothing happens. PySpark builds a logical plan. When you trigger an action (.show(), .write()), the Catalyst optimizer converts that logical plan to a physical plan, splits it into stages at shuffle boundaries, and distributes tasks across executors.

Shuffle write/read is the #1 performance bottleneck in 80%+ of slow Spark jobs. spark.sql.shuffle.partitions defaults to 200. Executor memory is split roughly 60% to the unified pool (spark.memory.fraction = 0.6). Understanding this model is what separates tutorial knowledge from interview readiness.