Apache Spark 4.0 and 4.1

Apache Spark 4.0 shipped in May 2025 with over 5,100 resolved tickets, and 2 of those changes will break your pipelines if you aren't paying attention.^[1] Spark ANSI mode is on by default, turning division by zero, integer overflow, and invalid casts from silent NULLs into hard runtime exceptions.^[2] Java 17 is the new minimum runtime; Java 8 and 11 are gone.^[1] Apache Spark 4.1 followed in December 2025, making the Spark VARIANT type generally available with shredding, promoting SQL scripting to GA, and anchoring Apache Iceberg 1.11.0 as the default build target.^[3] Alongside those: a 1.5 MB pyspark-client via Spark Connect, SQL user-defined functions, and pipe syntax for query composition. If you're a data engineer running Spark in production in 2026, the upgrade decision is no longer theoretical.

Spark ANSI Mode: Your Silent NULLs Are Now Loud Errors

This is the change that will bite the most teams. spark.sql.ansi.enabled flipped from false to true in Spark 4.0.^[2] In Spark 3.x, dividing by zero returned NULL. Casting the string "abc" to an integer returned NULL. Integer overflow wrapped around silently. All of that now throws exceptions.^[2]

Those silent NULLs were data quality bugs you never knew about because nothing exploded. A pipeline "working fine" for 3 years might have been producing wrong numbers for 3 years; you just never got an error to prove it. I've debugged Spark jobs that silently dropped records for months because of exactly this kind of quiet failure. Nothing crashed. The dashboards looked plausible. The numbers were just wrong.

The failures you see after enabling ANSI mode are data quality issues you already had. Every exception Spark 4.0 throws is a row that was silently wrong before.

You can set spark.sql.ansi.enabled=false to restore the old behavior, but that just reburies the problems.^[4] The better approach: use try_* functions to handle specific cases where you genuinely want NULL-on-failure semantics.

The smartest migration strategy I've seen: enable spark.sql.ansi.enabled=true in your Spark 3 environment before you upgrade. Surface the failures on the old runtime where you already know what "working" looks like. Every team I've talked to that skipped this step spent weeks debugging exceptions in Spark 4 that had nothing to do with Spark 4. They were latent bugs that finally had a voice.

Spark 3.5.x receives extended LTS support (security fixes only) through November 2027.^[5] That sounds like plenty of runway until you realize your Iceberg, Delta Lake, and Kafka connector versions all need to align with Spark 4 and Scala 2.13 builds. The dependency graph is the actual bottleneck.

Spark Now Requires Java 17

Spark 4.0 requires Java 17 as the minimum runtime. Java 21 is also supported. Java 8 and 11 are done, and Scala 2.12 is dropped entirely; only Scala 2.13 builds ship.^[1]

This sounds like a version bump. It's actually a dependency audit. Java 17's module system (Project Jigsaw) tightens encapsulation, which means reflection-heavy libraries like Netty and Jetty now need explicit --add-opens flags. The javax.servlet namespace is gone; it's jakarta.servlet now. Every connector, plugin, and custom UDF that touches JDK internals needs to be checked.

The practical blocker for most teams isn't Spark itself. It's the 3rd-party ecosystem. Your Spark-RAPIDS build, your Kafka connector, your Delta Lake version: they all need Spark 4 + Scala 2.13 compatible releases. Several enterprise shops discovered critical dependencies stuck on Scala 2.12 only after starting lab testing. Check your dependency tree before you write a single line of migration code.

Iceberg 1.11.0 also dropped Java 11, requiring Java 17 or 21 for its core libraries.^[6] The entire lakehouse stack is converging on Java 17 as the floor. If your cluster is still on Java 11, you're upgrading everything at once whether you planned to or not.

The Spark VARIANT Type: 8x Faster Than JSON Strings

Spark 4.0 introduced the native VARIANT data type for semi-structured data, and Spark 4.1 promoted it to GA with shredding support.^[3]

If you've been storing JSON payloads as string columns, you know the pain. Every field access means parsing the entire string. Every query re-parses it. get_json_object and from_json do string manipulation on every single row, every single time. VARIANT stores semi-structured data in an optimized binary encoding that eliminates that repeated parsing.

The numbers from Databricks (tested on Databricks Runtime 15.0 with Photon enabled): VARIANT delivers 8x faster reads compared to JSON strings. With shredding in Spark 4.1, that jumps to roughly 30x, though write performance trades a 20-50% slowdown for the improved read path. Storage efficiency improves by 22% over JSON strings.^[7]

The query syntax uses dot-notation, which is a significant ergonomic win over nested get_json_object calls:

VARIANT integrates natively with Apache Iceberg v3 tables, meaning your semi-structured columns get schema evolution, time travel, and row-level access control without upfront schema enforcement.^[8] For anyone building lakehouse architectures, this eliminates one of the oldest friction points between schema flexibility and query performance.

When shredding is enabled, frequently accessed fields get physically separated into typed columns with Parquet statistics, so Iceberg can prune whole files before reading any data. The write overhead is real, but for read-heavy analytical workloads on event data, logging pipelines, or any append-heavy table with frequent schema variation, the tradeoff is obvious. Know when VARIANT fits (high-cardinality nested fields, evolving schemas) and when typed columns are still the right call (stable schemas, complex transformations).

SQL Features That Actually Matter

Spark 4.0 and 4.1 shipped 3 SQL features worth knowing. Not because they're flashy, but because they change how you structure queries in production.

SQL User-Defined Functions

You can now define reusable functions entirely in SQL using CREATE FUNCTION, with no Scala or Python wrapper required.^[1] The functions integrate with Spark's query optimizer, which means the planner can see through them and optimize the execution plan. That's a meaningful difference from opaque Python UDFs that Spark treats as black boxes and serializes row-by-row.

Pipe Syntax

The |> operator lets you chain SQL transformations in reading order, top to bottom.^[1] If you've written (or debugged at 2am) a 200-line SQL query with 8 nested CTEs, you understand why this matters.

If you're already comfortable with CTEs and recursive queries (Spark 4.1 added recursive CTE support too^[3]), pipe syntax is the next ergonomic layer. Each step is isolated, testable, and readable by the person who inherits your query.

Session Variables and SQL Scripting

Spark 4.0 introduced DECLARE VARIABLE for session-scoped state.^[1] Variables are isolated per session, updatable via SET VAR, and usable wherever constant expressions are allowed. They can't be used in persisted views or generated columns, which is the right constraint. Pair this with SQL scripting reaching GA in Spark 4.1 (loops, conditionals, CONTINUE HANDLER for error recovery), and procedural logic that used to require a Python wrapper can live entirely in SQL.^[3]

The 1.5 MB pyspark-client

The full PySpark installation is 355 MB and requires a JRE. The new pyspark-client, shipped via Spark Connect in Spark 4.0, is 1.5 MB with zero Java dependencies.^[1] It communicates with Spark clusters over gRPC, sending unresolved logical plans as protocol buffers and receiving Arrow-encoded results.^[9] Pure Python. No JVM on your local machine.

The practical impact hits 3 places. CI/CD pipelines: your Docker images drop from 2+ GB to base Python plus 1.5 MB. Developer onboarding: new engineers run PySpark on an M1 Mac or Windows WSL with no local Spark install. Notebook environments: thin clients connect to production-scale clusters without the JVM overhead that makes local development painful.

AWS already supports Spark Connect on Amazon EMR Serverless for interactive PySpark development without client-side Spark installation. The architecture is the same pattern showing up everywhere: decouple the client from the execution engine. If your CI pipeline spins up 100+ jobs a day, shaving 2 GB off each base image saves real minutes and real money.

Iceberg 1.11.0 as Default Build Target

Apache Iceberg 1.11.0, released May 2026, made Spark 4.1 and Flink 2.1 the default build targets for the first time.^[6] That release stabilized deletion vectors, the Variant type, native geospatial support, and nanosecond timestamps from experimental to production-ready.^[10]

This coupling matters because upgrading to Spark 4.1 is no longer just a Spark decision. It's an Iceberg decision, a Java 17 decision, and a Scala 2.13 decision, all at once. The flip side: if you're already planning an Iceberg upgrade, Spark 4.1 is now the path of least resistance.

Iceberg 1.11.0 also introduces server-side scan planning for REST catalogs, shifting metadata traversal from the query engine to the catalog server.^[6] For teams managing large Iceberg deployments across medallion architectures, this reduces client memory consumption and enables server-side metadata caching. And the format version constraint is real: V2 engines cannot read V3 tables, so upgrading to V3 for deletion vectors and VARIANT shredding is a full-stack coordination problem. Plan accordingly.

What Spark 4.x Means for Data Engineer Interviews

The defaults changed, and interview questions are following.

ANSI mode is the most testable change. Expect scenarios like: "Your pipeline returned NULL for division-by-zero in Spark 3 but throws an exception in Spark 4. What happened? How do you fix it?" If you can't explain try_divide, try_cast, and the scoped configuration options, you'll struggle. This is basic migration literacy for anyone claiming Spark experience in 2026.

System design rounds are starting to probe the Spark + Iceberg coupling. "Walk me through upgrading a 200-query pipeline from Spark 3.5 to 4.1." The answer involves Java 17 dependency audits, ANSI compatibility testing on the old runtime, Scala 2.13 connector checks, and Iceberg format version coordination. That's a real upgrade plan; interviewers want to hear you've thought through the order of operations.

VARIANT comes up in data modeling discussions. When do you use a VARIANT column versus typed columns? The answer depends on schema stability: high-cardinality nested fields with frequent schema evolution go VARIANT; stable schemas with complex transformations still want typed columns. Knowing the 8x read performance advantage (30x with shredding) gives your answer quantitative weight.^[7]

If you're working through Spark interview questions, the 4.x releases are now fair game. But here's the thing: the concepts are what transfer. Understanding why ANSI compliance changes error semantics, why binary encoding outperforms string parsing, how client-server decoupling works in distributed systems. The specific try_cast syntax is something you look up once. The reasoning behind it is the skill that compounds.

Spark 3.5 LTS runs through November 2027.^[5] That gives you runway, but the ecosystem is already moving. Iceberg 1.11.0 defaults to Spark 4.1. New features land in 4.x only. The migration isn't an emergency today, but the engineers who've already done it (or can credibly plan one) are the ones getting hired at the top of the band. Start with the ANSI flag in your Spark 3 environment. Surface the failures. Fix the data quality bugs. Then upgrade. And practice the real problems at DataDriven.

References

1. Apache Spark, "Spark Release 4.0.0," May 2025. spark.apache.org
2. Apache Spark, "Migration Guide: SQL, Datasets and DataFrame." apache.github.io
3. Apache Spark, "Spark 4.1.0 released," December 2025. spark.apache.org
4. Apache Spark, "ANSI Compliance," Spark 4.0.2 Documentation. spark.apache.org
5. Apache Spark, "Versioning Policy." spark.apache.org
6. Google Open Source Blog, "Announcing Apache Iceberg 1.11.0," May 2026. opensource.googleblog.com
7. Databricks, "Introducing the Open Variant Data Type in Delta Lake and Apache Spark." databricks.com
8. Amazon Web Services, "Beyond JSON blobs: Implementing the VARIANT data type in Apache Iceberg V3." aws.amazon.com
9. Apache Spark, "Spark Connect Overview." spark.apache.org
10. Apache Iceberg, "Apache Iceberg 1.11.0 Release," May 2026. iceberg.apache.org