The spark-submit Config Surface

Concepts covered: Configuration Tuning

Every flag is a property

Each named flag of spark-submit is a shortcut for a Spark property, one of the spark.* keys that the Environment tab of the Spark UI lists for a running application. A property with no shortcut goes through --conf key=value, which you can repeat.

Which value wins

One property can be set in 3 places, and Spark settles them in a fixed order. A value set in code on the session builder beats a spark-submit flag or --conf, and a flag beats the cluster's spark-defaults.conf file. Here is 1 job's settings traced through all 3.

So the job gets 10 executors of 4 cores, which is 40 slots, and every stage after a boundary runs 800 tasks, which is 20 waves.

Read once at startup, or per query

Precedence decides which value Spark reads; the other question is when Spark reads it. Spark reads executor memory and cores, along with the executor count, when executors launch, and every executor in an application launches with the values the application started with. Most properties under spark.sql., including spark.sql.shuffle.partitions, are read each time a query is planned, so a new value applies from the next action on.

spark.conf.set("spark.sql.shuffle.partitions", 400)
daily = events.groupBy("event_date").count()
daily.collect()

spark.conf.set("spark.executor.memory", "8g")

The session builder fails more quietly. In a notebook where a session already exists, SparkSession.builder.config("spark.executor.memory", "8g").getOrCreate() hands back that existing session, and its executors keep the memory they started with.

Driver memory follows the same rule, with a twist from the previous section. In client mode the driver is the very process running your code, so Spark's documentation says not to set spark.driver.memory in code there: by then the driver's JVM has already started. Pass --driver-memory or put it in spark-defaults.conf. In client mode that heap also comes from the machine you launched on, so a laptop with 16GB of RAM can't host a 24g driver.

Read once, at startup

• spark.executor.memory
• spark.executor.cores
• spark.executor.instances
• spark.driver.memory
• spark.submit.deployMode

Read per query

• spark.sql.shuffle.partitions
• spark.sql.files.maxPartitionBytes
• Most other spark.sql. properties

Checking what applied

When a job behaves as if a setting never landed, open the Environment tab: it lists every property the application started with and the value that won. A value changed later with spark.conf.set lives in the session and doesn't appear there, so check that one through its effect, such as the task count of the first stage after the next boundary.

spark-defaults.conf sets spark.sql.shuffle.partitions to 200, spark-submit passes --conf spark.sql.shuffle.partitions=300, and the session builder sets 600. Right before the only action, the code runs spark.conf.set("spark.sql.shuffle.partitions", 100). With adaptive execution off, how many tasks run in the stage after the groupBy?

• It runs 300 tasks, since a spark-submit flag outranks anything set in code
• It runs 600 tasks, since the builder value wins and later changes are refused
• It runs 200 tasks, since the defaults file is read last and overrides the rest
• It runs 100 tasks, since spark.sql. settings are read as each query is planned

About this lesson

This section is part of the Execution Model: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.