MAP: Dynamic Key-Value Pairs

Concepts covered: Map Columns

Flexible Attributes Without Schema Changes

A MAP column stores key-value pairs where the keys are not predefined in the schema. A user preferences column might contain {'theme': 'dark', 'language': 'en', 'notifications': 'off'}. Each user can have different keys. Adding a new preference does not require ALTER TABLE.

MAPs are the escape hatch for data that does not have a fixed schema. Event properties, configuration settings, A/B test parameters, custom metadata. Anything where the set of keys varies per row and evolves over time.

The Tradeoff

MAP Column

• Flexible: any keys, no schema change
• Good for sparse, varying attributes
• No column explosion for 100+ optional fields
• Cannot be indexed or used for partition pruning

Typed Columns

• Fixed: schema change to add a field
• Good for frequently-queried attributes
• Can be indexed and used for partitioning
• Column explosion if many optional fields

The hybrid pattern: store the full event payload as a MAP or JSON column for flexibility, and extract frequently-queried fields as top-level typed columns for performance. The JSON column preserves everything; the typed columns make the hot-path queries fast.

About this lesson

This section is part of the Nested Data lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.