Semi-Structured Data: Beginner
You are 20 minutes into a SQL screen at a fitness app. The interviewer shares a table with 3 columns: event_id, received_at, payload.
'Every row is one app event. Give me event counts per user and event type for yesterday.'
You scan the column list for user_id and it is not there. It sits inside payload, in a sample value that starts {"user_id": "u_101", "event_type": "page_view"}, and nobody has said which engine this is or whether payload holds a JSON type or plain text. The common first move is GROUP BY user_id, which errors before a single row comes back.
She is watching for 3 things: whether you notice the fields live inside a value, whether you know your engine's way of reading them out, and whether you turn them into columns before you count anything.
What you will be able to do
How the question disguises itself
| column | type | sample value (event 4) |
|---|---|---|
| event_id | BIGINT | 4 |
| received_at | TIMESTAMP | 2024-03-14 09:12:03 |
| payload | JSONB | {"user_id": "u_101", "event_type": "page_view"} |
Here user_id and event_type exist only as keys inside payload. The database holds that value as one document and keeps no record of the keys within it. WHERE and GROUP BY operate on columns, as does a JOIN condition, so a key inside a document is out of their reach until a function reads it out and returns an ordinary value. Writing GROUP BY user_id against this table fails with column does not exist, and that error in the first minute is how many candidates discover the problem.
Where the first minute goes wrong
Extract first, then write ordinary SQL
WITH extracted AS (SELECTpayload ->> 'user_id' AS user_id,payload ->> 'event_type' AS event_typeFROM events)SELECTuser_id,event_type,COUNT(*) AS event_countFROM extractedGROUP BY user_id, event_type
The CTE pulls user_id and event_type out of the JSON once, and the GROUP BY counts the plain columns
Why the extraction lives in a CTE
- payload->>'user_id' written out in SELECT and GROUP BY, then again in every filter
- Dialect syntax tangled into the business logic
- A renamed key means finding and fixing every copy
- Testable only by running the full aggregate
- Each key read out once and given a column name
- The outer query reads like SQL over a flat table
- A renamed key or a new engine touches one CTE
- SELECT * FROM extracted LIMIT 5 shows the parsed rows first
Questions worth asking first
- ▸Which engine? Each of the 5 engines here (Postgres, Snowflake, BigQuery, MySQL, Spark) spells extraction differently, and naming yours is the first signal of fluency.
- ▸Is payload stored as JSON or as text? Text holding JSON may need
PARSE_JSON(payload)in Snowflake orpayload::jsonbin Postgres before any path works. - ▸Which keys does the question need, and how deep? A top-level key and one nested 3 levels down use the same function with a longer path.
Every engine reads a value out of JSON by following a path: a route from the top of the document to one value. Most engines write the path as a JSONPath string. $ is the root, .key steps into a key of an object, and [n] steps into a position of an array, counted from 0. Postgres spells each step as an operator and Snowflake has its own colon syntax, but all of them walk the route one step at a time, and each step only works if the value it lands on has the shape the next step expects.
Each step of $.items[0].product_id narrows the value: the object, then its array, then the first element, then one string.
Read the path $.items[0].product_id against the diagram. The root is an object, so .items is legal and lands on an array. An array has positions, so [0] is legal and lands on the first object in the cart. That object has keys, so .product_id lands on the string p_88. Each step is a question asked of whatever value the previous step returned.
| Path | What the steps do | Value on event 2 |
|---|---|---|
| `$.user_id` | key at the root | u_101 |
| `$.device.platform` | into device, then its platform key | ios |
| `$.device.app_version` | into device, then app_version | 7.2.0 |
| `$.items[0].product_id` | into items, first element, its product_id | p_88 |
| `$.items[1].qty` | into items, second element, its qty | 1 |
| `$.items[2].product_id` | items holds only positions 0 and 1 | NULL |
The same path in 5 engines
Postgres is the one that looks different. Its operators take one step each: -> with a key name steps into an object, -> with an integer steps into an array, and the chain reads left to right in the same order as the path. When the path is long, #>> takes all the steps at once as a text array: payload #>> '{items,0,product_id}' returns the same p_88. Snowflake starts with a colon after the column and continues with dots and brackets; ::STRING at the end casts the result.
Keys with a dash or a dot in the name need quoting inside a path string, because the dot would otherwise read as a step. MySQL takes $."user-agent" and Snowflake takes payload:"user-agent". Postgres needs nothing extra: payload->'user-agent' already passes the key name as a string, so every character in it is part of the name.
Path mistakes that cost time
- ▸Counting from 1.
$.items[1]is the second element; the first is$.items[0]in every engine above. - ▸Skipping a level.
$.platformreturns NULL because platform lives under device; the full path is$.device.platform. - ▸Text mid-path in Postgres.
payload->>'device'->>'platform'fails: the first->>returns text, and text has no keys to step into. - ▸Asking an array for a key.
$.items.product_idreturns NULL in most engines, since items holds positions and a key step needs an object. - ▸Wrong case. JSON keys are case-sensitive, so
$.User_Idmisses a key stored as user_id.
The follow-up: the last item
A common push once the first item works: 'carts have different lengths, so how do you get the last one?' A fixed index stops working, and the answer depends on the engine. Postgres jsonb accepts a negative index, payload->'items'->-1. MySQL paths accept $.items[last]. Snowflake takes a computed position through GET: GET(payload:items, ARRAY_SIZE(payload:items) - 1). Naming the engine feature in one sentence scores the point; the array length behind the Snowflake form is the subject of the last section.
Extract event 2's event_type in 2 ways and you get 2 different answers: "purchase", 10 characters including the double quotes, or purchase, 8 characters. The first is a JSON value: the engine cut a piece out of the document and handed it back still written as JSON, and JSON writes a string inside quotes. The second is SQL text: the engine unwrapped the JSON string into a plain value that compares equal to the literal 'purchase'.
| Engine | Returns a JSON value | Returns bare text |
|---|---|---|
| Postgres | `payload->'event_type'` | `payload->>'event_type'` |
| MySQL | `JSON_EXTRACT(payload, '$.event_type')` | `payload->>'$.event_type'` |
| BigQuery | `JSON_QUERY` or `JSON_EXTRACT` | `JSON_VALUE` or `JSON_EXTRACT_SCALAR` |
| Snowflake | `payload:event_type` (a VARIANT) | `payload:event_type::STRING` |
| Trino | `json_extract` | `json_extract_scalar` |
Spark's get_json_object offers only one form: it returns a string, with the quotes already stripped from string values. Every other engine has both forms. Only the function or operator name tells you which one you wrote.
Why the quoted form exists
You walk the path with the JSON-returning form. Extract $.device or $.items and there is no bare text to unwrap: the result is an object or an array, and the next step needs it to still be JSON. So the choice splits by job. Containers and intermediate steps stay JSON; the final scalar that a later clause filters or joins on comes out as text. Postgres builds that rule into its operators: -> on each step that lands on a container, ->> on the step that lands on the value you want.
The comparison that returns nothing
The quoted form is dangerous because it usually runs. In BigQuery, JSON_EXTRACT over a STRING payload returns a STRING, so WHERE JSON_EXTRACT(payload, '$.event_type') = 'purchase' is a legal comparison between "purchase" and purchase. It's false on every row, the query reports 0 purchases, and nothing errors. Postgres reaches the same place through a cast: (payload->'user_id')::text is the 7-character string "u_101", and a join on it matches no user.
| Postgres expression | Type | Value on event 2 | Characters |
|---|---|---|---|
| `payload->'event_type'` | jsonb | "purchase" | n/a |
| `(payload->'event_type')::text` | text | "purchase" | 10 |
| `payload->>'event_type'` | text | purchase | 8 |
| `payload->'amount'` | jsonb | 49.90 | n/a |
| `payload->>'amount'` | text | 49.90 | 5 |
Numbers have no quotes to strip
Quotes appear only on JSON strings. A JSON number or boolean prints the same in both forms, as does null, which is why both amount rows show 49.90. What differs is the type: one is jsonb and the other text. Neither is NUMERIC. SUM(payload->>'amount') fails in Postgres because there is no SUM over text. The habit is to extract with the text form and cast on the same CTE line, (payload->>'amount')::NUMERIC AS amount, so every later clause sees a number.
MySQL arrows and Snowflake variants
MySQL puts both forms behind arrows that resemble Postgres's but take a path string: payload->'$.event_type' is JSON_EXTRACT and returns "purchase", while payload->>'$.event_type' is JSON_UNQUOTE(JSON_EXTRACT(...)) and returns purchase. Moving between the 2 engines, say which one you mean, since payload->>'event_type' without the $. is valid Postgres and an error in MySQL. In Snowflake, payload:event_type is a VARIANT that a result grid prints with quotes; ::STRING at the end gives the column a string type for everything downstream.
The sentence that scores
Say why you chose the operator: 'I'm using ->> because I'm comparing to a string literal; -> would keep the JSON quotes.' When a filter or join comes back empty, the first suspect worth naming is a quoted JSON value being compared to bare text. Some interviewers plant this deliberately, writing a query with the JSON-returning function and asking why the count is 0.
LENGTH of the extraction on a row you know. For event_type on event 2, 8 means bare text and 10 means the quotes came along.- The type is enforced on write: inserting 'N/A' fails
- NOT NULL and CHECK (amount >= 0) are enforced by the database
- A misspelled name fails before the query reads a row
- Read as stored, already a number
- Listed in the catalog with its name and type
- Any value is accepted: 49.90, "49.90", "N/A" and true all insert
- No constraint applies unless someone writes one per key
- A misspelled path runs and returns NULL
- Extracted and cast in every query that uses it
- Known only to whoever reads sample documents
The typo that compiles
There is no catalog entry for a key either. information_schema.columns lists amount when it is a column; when it is a key, the only way to learn that it exists, how it is spelled and what values it holds is to read documents. With a 10-row sample in an interview, read the sample aloud before you write a path.
Bad values surface when you read
A typed column rejects a bad value at insert, so the producer finds out. A JSON key stores whatever arrives, and the problem appears later, in whichever query casts it. If one client sent "amount": "N/A", then (payload->>'amount')::NUMERIC fails the entire query in Postgres with invalid input syntax for type numeric, and the failing row might be one purchase in a full day of events. Snowflake and BigQuery have safe casts that return NULL for that row instead and let the query finish. In Snowflake the VARIANT has to become a string first, because TRY_CAST accepts only string input: write TRY_CAST(payload:amount::STRING AS NUMBER(10,2)) or TRY_TO_NUMBER(payload:amount::STRING, 10, 2). Applied straight to payload:amount, TRY_CAST raises an error instead of returning NULL. BigQuery's SAFE_CAST(JSON_VALUE(payload, '$.amount') AS NUMERIC) already starts from text, since JSON_VALUE returns a string.
Every read pays the extraction
The planner picks that plan because the received_at predicate exists, wherever it sits in the WHERE clause: writing the range first or last produces the same plan. Some interviewers test this directly by asking whether the date condition should move to the top. The answer that scores says position does not matter. What limits the scan is a selective predicate on a typed column that is indexed or partitioned, and EXPLAIN shows the index scan or partition pruning that proves it.
- ▸Types: a column rejects 'N/A' at insert; a JSON key stores it and the cast fails at read time.
- ▸Constraints: NOT NULL and CHECK apply to columns without anyone writing extra rules.
- ▸Errors: a misspelled column fails to run; a misspelled path runs and returns NULL.
- ▸Discovery: a column is listed in the catalog; a key is found by reading sample documents.
- ▸Cost: a column is used as stored; a key is extracted and cast on every read.
SUM(CASE WHEN payload->>'user_id' IS NULL THEN 1 ELSE 0 END) grouped by event type. A schema would have refused those rows; with JSON you have to go and count them.| Engine | Length of the items array | Event 2 |
|---|---|---|
| Postgres | `jsonb_array_length(payload->'items')` | 2 |
| MySQL | `JSON_LENGTH(payload, '$.items')` | 2 |
| BigQuery | `ARRAY_LENGTH(JSON_QUERY_ARRAY(payload, '$.items'))` | 2 |
| Snowflake | `ARRAY_SIZE(payload:items)` | 2 |
| Trino | `json_array_length(json_extract(payload, '$.items'))` | 2 |
| Spark | `json_array_length(get_json_object(payload, '$.items'))` | 2 |
Each function is handed the array itself, which ties back to the 2 forms of extraction. In Postgres the argument is payload->'items', the JSON form, because a container has to stay JSON. Pass payload->>'items' and the call fails, since jsonb_array_length takes jsonb and the text form is plain text.
| event_id | user_id | item_count |
|---|---|---|
| 2 | u_101 | 2 |
Why not unnest and count
- Event 2 becomes 2 rows, then folds back to 1
- An event with "items": [] produces no rows and drops out
- Other event columns repeat once per item until regrouped
- A GROUP BY over every event column to read one number
- Event 2 stays 1 row with item_count 2
- An empty array returns 0 and the event stays
- Event-level columns such as amount stay at one per event
- A single call you can put inside a CASE or an AVG as easily as in WHERE
Wrong counts that look plausible
- ▸
LENGTH(payload->>'items')counts characters: 68 for event 2's items text in Postgres, where the answer is 2. - ▸MySQL's
JSON_LENGTHcounts the keys of an object and returns 1 for a scalar, so a path that lands on$.devicereturns 2 instead of an error. - ▸Postgres
jsonb_array_lengthon an object or a scalar raises an error instead of returning a number. - ▸An event with no items key passes NULL to the function and gets NULL back; wrap it in
COALESCE(..., 0)if the question counts that as an empty cart.
Lengths compose with the rest of the query
Once item_count is an integer column in the CTE, everything relational works on it: AVG(item_count) for average basket size, SUM(item_count) for line items across all purchases, a CASE that buckets carts by size with breaks at 2 items and at 5. The path from the second section and the length function combine freely in the same CTE, as the query below shows.
- Name your engine and its extraction function before writing the first line
- Extract each key once into a named, cast column inside a CTE
- Use the text-returning form for every final value your query compares or groups by
- Pass the JSON form of an array to the engine's length function
- Select from the extraction CTE on a few rows before aggregating
- Don't compare a JSON-returning extraction to a string literal
- Don't write [1] when you mean the first element of an array
- Don't explode an array into rows only to count its elements
- Don't call LENGTH on an array's text; it counts characters
- Don't trust a column of NULLs until the path spelling is checked
> A food delivery company keeps orders in a table with order_id, placed_at and an order_json column holding the restaurant, the customer and a dishes array. The operations lead asks for yesterday's orders with 4 or more dishes, counted by restaurant city, and wants the result joined to the team's cities dimension for regional managers.
Spot the fields hiding inside a JSON column, walk the path to them, and pull out values SQL can compare.
- Category
- SQL
- Difficulty
- beginner
- Duration
- 25 minutes
- Challenges
- 4 hands-on challenges
Topics covered: When the Schema Is Inside the Data, JSON_EXTRACT Path Navigation, Quoted JSON Values vs Bare Text, JSON Columns vs Flat Columns, Counting Elements in a JSON Array
Lesson Sections
- When the Schema Is Inside the Data (concepts: JSON Extraction, Common Table Expressions, Grouping, Counting)
How the question disguises itself The request almost never contains the word JSON. It names a field instead: count events per user, average order value by platform, crashes by app version. Then the table on screen has a column called payload or properties (sometimes attributes, context, metadata, raw), and the field you need is nowhere in the column list. That gap is the tell. When a noun from the question has no matching column, read the sample rows, because the field is almost always sitting i
- JSON_EXTRACT Path Navigation (concepts: JSON Extraction)
Here is event 2 in full. The animation in the previous section showed only its first 2 keys; real purchase payloads also carry an amount alongside 2 nested structures, a device object and an items array. The same path in 5 engines Path mistakes that cost time The follow-up: the last item
- Quoted JSON Values vs Bare Text (concepts: JSON Extraction, Type Casting, Inner Joins)
Why the quoted form exists The comparison that returns nothing Run both against the 4 events from the first section and a users table holding u_101 and u_102. The first query returns 0. The second returns 4, one match per event. The only difference between them is a single character in the operator. Numbers have no quotes to strip MySQL arrows and Snowflake variants The sentence that scores
- JSON Columns vs Flat Columns (concepts: JSON Extraction, Basic Types, Safe Casting, Null Handling, Storage Optimization)
This one usually arrives as a follow-up once your extraction runs: 'That payload column is flexible. Why don't we store every table like that?' The interviewer is checking whether you know what a column does for you that a key inside a document cannot. The answer that scores names the specific losses, each with a concrete example from the table in front of you. The typo that compiles A misspelled column fails fast. A misspelled path does not fail at all: the engine cannot know which keys a docum
- Counting Elements in a JSON Array (concepts: JSON Extraction, Sums & Averages)
The request sounds like an aggregation: 'how many items were in each order', 'find carts with 3 or more products', 'what's the average basket size'. The reflex is to explode the array into one row per element and count them back up. You don't need to, because a JSON array already knows its own length, and every major engine has a function that returns it as an integer on the same row. The purchase went in as one row and came out as one row, and the count came from a single function call. The que