Iceberg 1.11 Scan Planning

Apache Iceberg 1.11.0 shipped on May 19, 2026, according to the project's releases page.^[1] The ASF Release Catalog lists May 15 instead and notes that its entry hasn't been reviewed by the Iceberg project yet, so the exact day is slightly unsettled.^[2] The headline feature is remote scan planning in the Iceberg REST catalog. The catalog server plans the table scan and streams file scan tasks back to the engine, so the client no longer reads manifest lists and manifests itself.^[3] The project describes the REST changes as the most significant step for that protocol since it was introduced. The release has over 1,000 commits from 200+ contributors.^[3]

It also ships built-in table encryption, a pluggable File Format API, a partition statistics scan API, and, per Google's announcement, Spark 4.1 and Flink 2.1 as default build targets.^[4] Our earlier Iceberg v3 coverage went through table-format features like deletion vectors and Variant. This piece covers the catalog and engine changes.

My take: 1.11 moves query planning from the engine to the catalog, and that changes what you need to be able to explain in a system design loop. The old answer, "the engine reads metadata and figures it out," is now only half of the picture.

I've watched a Spark driver fall over while planning a scan on a table with years of small files, before it read a single row of data. Everybody stared at the executors. The driver was the problem. That class of problem is what this release goes after.

What Apache Iceberg 1.11 Actually Shipped

The facts first, with the source for each:

• Remote scan planning in the REST protocol. The catalog plans scans and returns file scan tasks. This covers incremental scans for Structured Streaming and metadata tables such as history and snapshots.^[3]
• Per-table opt-out. A table-level setting lets individual tables opt out of the catalog-level scan planning mode.^[3]
• Retry-safe REST plumbing. ETag-based table loading that returns 304 Not Modified, and an Idempotency-Key header for retried writes.^[3]
• File Format API. A pluggable FormatModel interface replaces hardcoded per-format handling. Parquet, ORC, Avro, and Arrow each implement it.^[3]
• Table encryption, a partition statistics scan API, and Spark 4.1 and Flink 2.1 as default build targets, as reported in Google's write-up of the Apache release.^[4]

One caveat on sourcing. Some of these details, especially the engine defaults, come from Google's Open Source Blog post about the release and not from the Apache release notes. I couldn't confirm the default-target wording in the project's own notes. Treat that line as reported, and check the GitHub release tag before you pin anything in production.

One more: this is October 2026. Check the releases page before you assume 1.11.0 is still the newest version. Point releases come out quickly.

Iceberg Scan Planning Moves Into the REST Catalog

Here's how client-side planning works. Your engine loads table metadata, reads the manifest list, reads the manifests it needs, prunes files against your predicates, and builds the task list. All of that happens in the client process, and on Spark that means the driver. On a big table with a messy history, that's a lot of metadata I/O and memory before any real work starts.

With server-side planning, the engine sends the scan to the catalog and gets file scan tasks back. Google describes the flow as one POST plan request per scan. Small scans return right away, larger ones return a plan ID that the client polls, and very large ones fetch tasks in parallel.^[4]

How You Turn It On

The planning mode is a catalog property, scan-planning-mode, with values client (the default) and server. The server can override the mode per table in its load-table response.^[5] Capitalization of the values has differed between doc versions, so match the docs page for the version you actually run.

Setting the flag doesn't mean you get server-side planning. The catalog also has to implement and advertise the planning endpoints, and your engine has to know how to call them. Engine and catalog support varies by vendor right now. Find out what your stack actually does by reading its docs, then confirm it by watching the requests.

What Changes Operationally

This is my analysis, not a claim from the release notes. Moving planning to the catalog trades one bottleneck for another:

• The client gets lighter. Less metadata transfer, less driver memory, faster job startup on big tables.
• The catalog gets heavier. It's now a planning service with capacity limits, latency variance, timeouts, and state for in-flight plans. If 40 jobs start at 6am, the catalog takes all 40 planning loads at once.
• The catalog sees more. It now knows your predicates and decides which files a query gets. That makes it a natural place to enforce and audit access, and also a much bigger blast radius when it's down.

The ETag and Idempotency-Key additions point the same direction. A catalog that can answer "nothing changed" with a 304, and that can recognize a retried commit, is being built like a real service. If you've ever had a retry double-write a partition, you already know why idempotency keys matter. Our idempotent pipeline design page covers the same idea at the pipeline layer.

Key takeaway: In 1.11, "where does query planning run?" has a configurable answer. You can now design the catalog as a shared planning service with its own capacity, latency, and failure modes, and interviewers at senior levels will expect you to reason about it that way.

Iceberg Table Encryption: What's Covered and What Leaks

The official encryption docs are specific, which I appreciate. Iceberg encrypts data files, delete files, manifests, and manifest lists. The metadata.json file stays unencrypted because it holds no data or stats.^[6]

The scope limits matter more than the feature list:

• Encryption currently works only with the Hive and REST catalogs.^[6]
• It covers only Parquet and Avro data formats.^[6] If your tables are ORC, this isn't for you yet.
• Activation takes 2 settings: a catalog property selecting the KMS (a built-in client or a custom class), and a table property, encryption.key-id, naming the master key in the KMS. Catalogs must not change or remove that key ID during the table's life.^[6]
• Each data and delete file gets its own data key and AAD prefix, stored in the parent manifest's key_metadata field.^[6]

Google's announcement adds that each data file and manifest list is encrypted with AES-GCM under its own data encryption key.^[4] Secondhand summaries of the 1.11.0 post also mention automatic key-encryption-key rotation and a unified KMS property across clouds. I couldn't confirm the exact wording of those, so I'm not giving you property names for them. Use the docs page for your version.

Using the documented table property, a new encrypted table looks like this:

Now my opinion on the trade-off. A plaintext metadata.json means the table's structure stays readable to anyone with storage access: the schema, the partition spec, and pointers into the snapshot tree. Your rows are protected. Your table's shape is not. For most teams that's fine. For a team whose column names are themselves sensitive (yes, I've seen is_under_investigation in a production schema), it's a conversation to have with security before launch, not after.

The chain to know by heart: the KMS holds the master key, the writer generates a data key per file and wraps it, and the manifest stores the wrapped key. That's envelope encryption, and it's one of the most common governance follow-ups in a design loop.

The File Format API and the Partition Statistics Scan

File Format API

Before 1.11, format handling was hardcoded per format. In 1.11 a FormatModel contract sits in between, and Parquet, ORC, Avro, and Arrow each implement it.^[3] Core Iceberg talks to an interface and stops caring which format is underneath.

This is plumbing, and good plumbing is boring. The practical result is that a new file format can plug in without every engine integration growing another branch of format-specific code. The cost is that every engine now depends on one shared interface. When it changes, everyone moves together.

One precision point. This is a change to the Java implementation. If your stack includes non-Java clients, don't assume they picked up the same abstraction. Interviewers love catching "Iceberg supports any format now" with a follow-up about which library.

Partition Statistics Scan

Small but concrete: PartitionStatsHandler.readPartitionStatsFile is deprecated in 1.11.0, scheduled for removal in 1.12.0, and the Javadoc points to PartitionStatisticsScan as the replacement.^[7] If you have internal tooling reading partition stats, that's a migration ticket. Write it now, while it's 2 lines and not a broken job during an incident.

Spark 4.1 and Flink 2.1 Become Default Targets for Data Engineers

Per Google's write-up of the release, 1.11.0 adds Spark 4.1 and Flink 2.1 support and makes both default build targets. Spark's MERGE gains a WITH SCHEMA EVOLUTION clause, and the Spark connector moves to Spark's newer DataSource V2 APIs.^[4] The same post describes Flink 2.1 work as initial, so I wouldn't put it in a critical streaming path on day one.

"Default build target" means what the project builds and tests by default. It doesn't mean older engines stopped working, and it doesn't mean the new ones are mature everywhere. Keep "default," "supported," and "deprecated" separate in your head. Upgrade questions in interviews probe exactly that distinction.

The operational lesson is the one every Data Engineer learns the hard way: the Iceberg runtime jar must match your engine's major version and Scala version. A mismatch usually fails at class-load time, not at build time, so your CI goes green and your 2am job goes red. Run old and new runtimes side by side in staging, check the catalog version, then cut over. If you need a refresher on how Spark loads catalogs and extensions, our Spark interview questions page covers it.

What Data Engineers Should Explain in System Design Interviews

I've sat on enough panels to know how this goes. A candidate draws S3, then Iceberg, then Spark, then Trino, and arrows everywhere. Then someone asks "where does the catalog fit?" and the whiteboard goes quiet.

The catalog is now the control plane: namespaces, authorization, credential vending, and, as of 1.11, planning. Concepts over tools, as always. The concept is "which layer owns which decision," and Iceberg 1.11 just made the answer richer. Here's what I'd practice:

If lakehouse vocabulary itself is shaky, start with data lake vs warehouse vs lakehouse before you start memorizing REST endpoints. Then work through the pipeline-architecture prompts in our Data Engineering system design guide, because that's where these catalog questions actually show up.

The candidates who stand out answer "it depends" and then say exactly what it depends on. Knowing that server-side planning needs the catalog to advertise the endpoints, and that engine support varies by vendor, shows you read primary docs instead of vendor marketing. Interviewers notice that.

What Data Engineers Should Do Next

Based on what's documented, here's the order I'd go in:

• Check your catalog first. Does it advertise scan-planning endpoints? If not, scan-planning-mode=server does nothing useful for you yet.
• Pick one painful table. The one with the slow-planning, driver-hungry scan. Trial server-side planning there, and measure job startup time and catalog latency before and after.
• Load-test the catalog. If it's now a planning service, treat it like one: concurrency at peak schedule time, timeouts, and what happens to your jobs when it's slow.
• Scope encryption honestly. Hive or REST catalog, Parquet or Avro, a key ID that will never change for the table's life. If any of those fail, you're waiting.
• File the deprecation ticket for readPartitionStatsFile before 1.12 removes it.
• Pin runtimes deliberately when moving to Spark 4.1 or Flink 2.1. Stage both, then cut over.

For interview prep, say it out loud. Explaining client vs catalog planning to a whiteboard is a different skill from configuring it. The best practice I've found for that is a timed mock interview where someone pushes back on your design.

Watch 2 things over the next few releases: how many engines and managed catalogs actually ship server-side planning, and how catalog operators handle the extra load. The format layer got most of the attention for years. The catalog is where the interesting work is now, and it's where the 3am pages will come from next.

References

1. Apache Iceberg project, "Releases," accessed October 2026. iceberg.apache.org
2. Apache Software Foundation, "Release catalog: Iceberg 1.11.0," accessed October 2026. release-catalog.apache.org
3. Apache Iceberg project, "Apache Iceberg 1.11.0 Release," May 2026. iceberg.apache.org
4. Google Open Source Blog, "Announcing Apache Iceberg 1.11.0," May 27, 2026 (reporting on the Apache Iceberg 1.11.0 release). opensource.googleblog.com
5. Apache Iceberg project, "Catalog properties," documentation (latest). iceberg.apache.org
6. Apache Iceberg project, "Encryption," documentation (latest). iceberg.apache.org
7. Apache Iceberg project, "PartitionStatsHandler," 1.11.0 Javadoc. iceberg.apache.org