Web Scraping Data Type Problems and Solutions

Scrapers return nested JSON, broken dates, and currency strings. How data engineers handle them, and how to practice the same problems in Python.

DataDriven Field Notes

Updated July 16, 20268 min readBy DataDriven Editorial

What this post covers

01How data engineers tame scraped data: Schema validation, type coercion, null handling, deduplication, normalization pipelines
02Data formats DE interviews keep asking about: JSON vs CSV vs Parquet vs Avro, columnar vs row, serialization trade-offs
03The data types web scraping actually returns: Nested JSON, JSON-LD, HTML fragments, dates, currencies, encodings, inconsistent schemas
04The same data type problems show up in Python screens: Flattening nested dicts, date parsing, string to number coercion, null handling
05Practicing data type problems deliberately: Graded Python practice problems with feedback beat ad-hoc scripting

I once spent 3 days debugging a pipeline that was silently converting every price on an e-commerce site to an integer. "$29.99" became "2999" because someone stripped the currency symbol and the decimal in the same regex. No errors. No warnings. Just a pricing table where everything cost 100x what it should. That's web scraping data types in a nutshell: the scraper runs, the data lands, everything looks fine, and downstream analytics are completely wrong.

The messy data that scrapers produce isn't some niche problem. It's the exact category of problem that data engineering interviews test, that Python screens gate on, and that separates junior engineers from senior ones. If you can wrangle scraped data, you can pass a technical screen. The skills are the same.

Prepare for the interview

01 / Open invite

02min.

Know web scraping data types the way the interviewer who asks it knows it.

a web scraping data types query, the same shape a screen would give you.

The diff against expected. Where ties broke. What you missed.

sandbox

1def sessionize(events):

2 sessions = []

3 for e in events:

4 if gap_minutes(e) > 30:

Execute your solution0.4s avg.

PayPalInterview question

Solve a web scraping data types problem

The Data Types Web Scraping Actually Returns

Scrapers don't return clean tables. They return a grab bag of formats that each break in their own special way.

Nested JSON is the most common. API responses, JavaScript-rendered pages, and single-page apps all serve deeply nested objects where the field you need is 4 levels deep inside an array of dicts. The schema changes without warning because the site redesigned a component, and now your product.offers.price path returns None on 300 of 10,000 pages.

JSON-LD is the best-case scenario when it exists. It's structured data embedded in <script type="application/ld+json"> tags, and it often contains exactly what you need without CSS selectors. But JSON-LD on real websites is frequently malformed: unquoted keys, invalid syntax, missing critical fields. The Dolce & Gabbana website ships a "highly detailed" product schema that's missing the price field entirely.

Date strings are ambiguous by default. "01/02/2026" is January 2nd in the US and February 1st in Europe. Without ISO 8601 normalization, you get off-by-one errors that compound silently across millions of records.

Currency strings mix symbols, commas, periods, and spaces in ways that break naive parsing. "$1,000.50" and "€1 000,50" represent the same concept but require completely different handling.

HTML fragments sneak into text fields: encoded entities, stray tags, broken Unicode. And encoding issues (mojibake, mixed UTF-8 and Latin-1) must be caught at scrape time. Converting garbled text downstream wastes pipeline cycles and creates persistent problems that leak into storage.

The worst part isn't the failures. It's the silent correctness drift. A page loads, a CSS selector matches, fields populate, the row is stored. The system reports success. But only partial data was captured, and downstream analytics run on incomplete records.

How Data Engineers Handle Scraped Data Cleaning

The playbook for scraped data cleaning has 3 layers: validate the schema, normalize the values, deduplicate the records. Skip any layer and you're building on sand.

Schema validation with Pydantic is the modern standard. Pydantic V2 defaults to Lax Mode, which coerces messy web strings ("123.45" to float, "True" to bool) without the performance penalty of strict mode. V2.10+ supports partial validation for streaming scrapers that receive truncated JSON, recovering valid objects instead of discarding entire batches. Strict mode skips type coercion and breaks on the string-heavy reality of web data. Lax mode is what you want.

Hash-based deduplication is O(1). Concatenate all field values, generate SHA-256, compare hashes. No sorting, no nested loops. This scales to any dataset size because hash comparisons are constant time.

But here's what most people miss: normalization and deduplication are not independent. Normalization alone (date standardization, currency consistency) without deduplication leaves duplicate entities. Deduplication alone without normalization lets duplicates slip through because formats don't match. You need both, in that order.

Most scraping projects fail silently, not because of missing data, but because of bad data that goes unnoticed. One wrong field distorts entire downstream analyses.

Gartner puts the average cost of poor data quality at $12.9 to $15 million per year per organization. 77% of organizations rate their data quality as average or worse. And 59% don't even measure it. These aren't abstract numbers; they're the direct consequence of pipelines that ingest scraped data without validation gates.

The Same Data Type Problems Show Up in Python Screens

Here's where this gets directly relevant to your career. The messy data scrapers produce is exactly what Python technical screens test. Not because interviewers love web scraping, but because these problems test whether you've touched production data.

Nested JSON Interview Questions

Nested JSON interview questions are asked at Amazon, Netflix, Meta, and basically every company with a data engineering role. The problem: given a nested object with arrays, flatten it to a single-level dictionary. {"a": {"b": "c", "d": "e"}} becomes {"a_b": "c", "a_d": "e"}.

	def flatten_json(obj, parent_key="", sep="_"):
	items = {}
	for k, v in obj.items():
	new_key = f"{parent_key}{sep}{k}" if parent_key else k
	if isinstance(v, dict):
	items.update(flatten_json(v, new_key, sep))
	elif isinstance(v, list):
	for i, item in enumerate(v):
	if isinstance(item, dict):
	items.update(flatten_json(item, f"{new_key}{sep}{i}", sep))
	else:
	items[f"{new_key}{sep}{i}"] = item
	else:
	items[new_key] = v
	return items

The function itself isn't hard. What separates candidates is edge case handling. What happens when v is None? When the list contains mixed types? When nesting goes 8 levels deep and you hit recursion limits? Interviewers at Meta and Amazon specifically probe these failure modes.

Date Parsing and Type Coercion

Python has no implicit type coercion. Unlike JavaScript, where "5" + 3 gives you "53", Python forces you to call int(), float(), or str() explicitly. This is intentional: automatic coercion is a source of silent bugs. But it means converting non-numeric strings raises ValueError, and that's a common crash point in screening.

	from datetime import datetime

	def parse_price(raw):
	if raw is None:
	return None
	# Strip currency symbols and whitespace
	cleaned = raw.strip().replace("$", "").replace("€", "").replace(",", "")
	try:
	return float(cleaned)
	except ValueError:
	return None

	def parse_date(raw, formats=None):
	if raw is None:
	return None
	formats = formats or ["%Y-%m-%d", "%m/%d/%Y", "%d/%m/%Y", "%B %d, %Y"]
	for fmt in formats:
	try:
	return datetime.strptime(raw.strip(), fmt)
	except ValueError:
	continue
	return None

Notice the pattern: check for None first (with is None, not == None), attempt the conversion, catch the exception, return a sensible default. This is the pattern phone screens are testing. 80 to 90% of candidates fail 2026 coding interviews, and data type mishandling is a primary culprit.

Phone screens for data engineers focus on data cleaning, not algorithms. Missing values, invalid date formats, currency symbols in numeric columns, duplicate rows. These mirror real scraper output directly. Courses will teach you theory you already know. What you need is reps on the stuff that's tripping you up in interviews.

The Fuller Copy

> A scraping pipeline pulls product listings from several sites, and each raw record differs by source: a price may sit under the first entry of `offers` or as a top-level field, and it arrives as a string like `"$1,299.00"`, while the id and title hide behind different key names. Flatten each raw record into a uniform listing, coercing prices to floats and timestamps to a plain date, and leaving any field that is absent as `None`. The same listing can be scraped from more than one page under the same id, so when an id repeats, keep the copy with the fewest missing fields.

Sample input & expected output(2 examples)

Input · example 1

records:

[
  {
    "id": "P100",
    "name": "Aeron Chair",
    "offers": [{"price":"$1,299.00","priceCurrency":"USD"}],
    "datePosted": "2026-07-01T12:00:00Z",
    "source_site": "chairhub"
  },
  {
    "id": "P200",
    "price": "$499.50",
    "title": "Standing Desk",
    "datePosted": "2026-06-15T08:30:00Z",
    "source_site": "deskmart"
  },
  {"sku":"P100","price":1299,"title":"Aeron Chair","source_site":"chairhub"}
]

Output

[
  {
    "id": "P100",
    "price": 1299,
    "title": "Aeron Chair",
    "currency": "USD",
    "posted_at": "2026-07-01",
    "source_site": "chairhub"
  },
  {
    "id": "P200",
    "price": 499.5,
    "title": "Standing Desk",
    "currency": null,
    "posted_at": "2026-06-15",
    "source_site": "deskmart"
  }
]

Input · example 2

records:[{"id":"P300","name":"Desk Lamp","source_site":"lightco"}]

Output

[
  {
    "id": "P300",
    "price": null,
    "title": "Desk Lamp",
    "currency": null,
    "posted_at": null,
    "source_site": "lightco"
  }
]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99



def normalize_listings(records):
    
  return
def normalize_listings(records):
    
  return

Data Engineer Interview Data Formats

Data engineer interview data formats come up in every pipeline design question. The interviewer says "how would you store this data?" and your answer reveals whether you've shipped pipelines or just read about them.

The core trade-off is columnar vs. row-oriented. Parquet achieves 60 to 75% storage reduction compared to row formats while accelerating analytical queries 3 to 5x. Athena queries scanning 2 columns from a wide dataset read an order of magnitude less data with Parquet than JSON. Avro enables seamless schema evolution with backward and forward compatibility, making it standard for write-heavy Kafka pipelines.

The hidden test: when you say "use Parquet," does the interviewer ask "why not Avro?" and can you articulate the difference? Avro's strength is streaming and schema evolution (write-heavy). Parquet's is analytical queries and compression (read-heavy). Confusing these is a red flag.

JSON files are 1.5 to 3x larger than CSV for the same tabular data, but when compressed, the difference shrinks to 10 to 20%. Protobuf binary serialization reduces message size 30 to 70% vs. JSON. These aren't trivia; they're the economics of pipeline design. Storage is cheap, but egress and scan costs aren't.

Strong answers sound like this: "Parquet for cold storage and analytics, Avro for event streams on Kafka, JSON only for public APIs because it's human-readable." Then you mention compression trade-offs: zstd achieves up to 40% smaller file sizes than Snappy, but the CPU cost of decompression on every read matters. That trade-off, storage vs. compute, is exactly what production systems debate, and naming it signals maturity.

Netflix values engineers who ask 2 to 3 sharp clarifying questions about requirements before proposing a format. That behavior differentiates you from 60% of the candidate pool. Rather than memorizing Parquet specifications, focus on understanding when and why to use it, and discuss trade-offs with other formats. The interview is testing whether you'd be useful on a 2am production incident, not whether you've memorized file format specs.

Practicing Data Type Problems Deliberately

Here's the uncomfortable truth: professional FAANG engineers solve LeetCode Medium problems at a 52.5% success rate during timed practice. If staff-level engineers are failing half the time under pressure, raw talent isn't enough. You need reps.

But not just any reps. Spaced repetition with repeated practice retains 74% more information vs. cramming. And 85% of candidates in structured peer-feedback practice courses succeeded in subsequent interviews. The data is clear: graded python practice problems with feedback beat ad-hoc scripting in a Jupyter notebook.

What Deliberate Practice Looks Like

Generic algorithm puzzles are fine for SWE roles. For data engineering, you want problems that look like real work: parse a messy log, dedupe records on a composite key, flatten a nested JSON blob, handle schema drift across 10,000 source files. These are the scenarios that DE interview prep should focus on.

The 4 data type problems that gate advancement:

Flatten nested dicts recursively with proper None and empty-list handling
Parse dates with strptime() across multiple ambiguous formats
Coerce strings to numbers safely with currency symbols, commas, and locale-specific decimals
Handle None with is checks throughout, not equality comparisons

20 graded, hard-feedback problems over 8 weeks outperform 500 untimed, no-feedback exercises. The feedback is the point. "Your flattening logic doesn't handle null arrays correctly" corrects a misconception before it calcifies. Running code in a REPL and eyeballing the output doesn't give you that signal.

Why AI-Assisted Practice Doesn't Count

AI-generated code has 1.7x more critical issues and 1.4x more major issues than human code, largely around unhandled edge cases and type assumptions. That's the exact failure mode we're talking about. If you're using an LLM to generate your practice solutions, you're rehearsing the gaps instead of closing them.

Research on AI-driven feedback systems found that "metacognitive laziness and overreliance from blindly accepting GenAI feedback without task analysis have begun to erode learners' academic achievements." Translation: if you let the AI debug your type coercion, you haven't learned type coercion.

Schema drift is the hidden differentiator in screens. Candidates who name-drop "schema validation," "dynamic typing," or tools like Great Expectations during type discussions pass. Those who optimize memory usage on toy datasets fail. Real screening separates engineers who've felt production pain from those who haven't.

The Skill That Connects All of This

Web scraping data types, null handling, format selection, type coercion: these aren't 4 separate topics. They're one skill. The ability to look at messy input and reason about what could go wrong before it goes wrong. That's what interviewers are actually testing, even when the question is "flatten this JSON" or "why Parquet over CSV."

Gartner predicts 60% of AI projects will be abandoned through 2026 due to lack of AI-ready data. 73% of enterprise data leaders rank data quality, not model engineering, as the bottleneck. The industry needs engineers who can wrangle messy data. The interview process, clumsy as it is, is trying to find those engineers.

Interviewing is a skill. It's separate from the actual job. Treat prep like a job. But for data engineers, the gap between interview prep and real work is smaller than you think. The scraped data problems you solve in production are the same problems you solve in a screen. Get your reps in on the real thing, with real feedback, and the screen becomes a formality.

web scraping data typesscraped data cleaningpython practice problemsdata engineer interview data formatsnested json interview questions

02 / Why practice

Try the actual problems

01
Active recall beats re-reading by 50%
Cognitive-science meta-reviews (Dunlosky et al., 2013) rank practice testing as a top-tier study technique, while re-reading and highlighting rank near the bottom
02
76% of hiring managers reject on the coding task, not the resume
From HackerRank's 2024 Developer Skills Report. Candidates who look strong on paper still fail the live screen if they haven't done timed, executable practice
03
Five problem shapes cover 80% of data engineer loops
Parsing and reshaping, sessionization, dedup with tie-breaks, streaming aggregation, top-N-per-group. Writing them by hand turns the unfamiliar into pattern recognition

Start practicing

Related interview prep

senior data engineer interview guide→

Senior Data Engineer interview process, scope-of-impact framing, technical leadership signals.

FAANG data engineer interview questions→

Real questions from Meta, Amazon, Apple, Netflix, and Google Data Engineer loops, with answers.

system design round prep guide→

Pipeline architecture, exactly-once semantics, and the framing that gets you to L5.

←All articles