What goes wrong with a mutable default argument?
def f(x, acc=[]) evaluates the default once at definition time, so every call without acc shares one list and results accumulate across calls. The fix is acc=None with an inside-the-function default. Interviewers plant this in code-review prompts because it looks correct and fails statefully.
Is dict ordered in Python?
Insertion-ordered since 3.7, guaranteed by the language. The distinction that matters: iteration order is insertion order, not sorted order and not frequency order, so a counting dict does not come back most-common-first. OrderedDict still earns its keep for move_to_end and popitem(last=False), which is why LRU implementations reach for it.
When do you use a generator instead of a list?
When the data does not need to exist all at once: streaming a large file, chaining transforms, or short-circuiting on the first match. A generator holds one element of state; a list holds everything. In pipeline code the difference is whether a 10 GB input fits in memory, which makes this a favorite data engineering follow-up.
Why are sets the answer to most membership questions?
x in list scans; x in set hashes. Inside a loop that turns O(n) into O(n squared) versus O(n). The reflex to demonstrate: any repeated membership check against a collection you built yourself belongs in a set, and the conversion cost is paid once.
What does sorted()'s key parameter buy you, and is the sort stable?
key maps each element to its sort value once, and Python's sort is stable, so equal keys keep their original order. Composite contracts like count descending then word ascending compress into one key tuple, (-count, word). Stability also means you can sort twice for layered orderings, minor key first.
Shallow copy versus deep copy?
A shallow copy duplicates the outer container and shares every nested object; deepcopy recurses. The practical bite: copying a list of dicts shallowly and mutating a row mutates the original. In pipeline code the safer habit is building new structures instead of copying and mutating at all.
What are Counter and defaultdict for?
Counter is a dict subclass for tallies with most_common built in; defaultdict removes the missing-key branch from grouping code. Both signal stdlib fluency in an interview, and both have a one-line handwritten equivalent (get with a default, setdefault) worth knowing when the interviewer asks you to avoid imports.
is versus ==?
== compares values; is compares identity. The only routine correct use of is in application code is None checks (x is None). Small-integer and string interning make is appear to work on values in quick experiments and then fail in production, which is exactly why the question gets asked.
Does the GIL matter for data engineering Python?
Less than the question implies. The GIL serializes CPU-bound threads, but pipeline code is usually I/O-bound (threads fine) or hands the heavy lifting to C libraries and engines (pandas, polars, Spark) that release it. The senior answer names the boundary: multiprocessing or a distributed engine for CPU-bound transforms, threads or async for I/O fan-out.
When do you reach for pandas in an interview, and when not?
If the prompt is genuinely tabular (group, join, pivot on rows and columns), pandas is fine and fast to write. If the prompt is about structures, streams, or state, stdlib answers read stronger and avoid version trivia. Ask the interviewer whether libraries are in bounds; the question itself is a good scoping signal.
How do you process a 100GB CSV on a machine with 16GB of RAM?
Stream it rather than load it. In pandas that is read_csv with a chunksize, processing each chunk and accumulating only the aggregate you need. In pure Python it is iterating the file object line by line, which is already lazy. If the transform needs shuffles or joins across the whole set, that is the signal to move to a distributed engine like Spark rather than to a bigger machine.
What are generators and the yield keyword, and why do they matter for big data?
A generator function returns an iterator that produces values one at a time and suspends its state between them, so it holds one element in memory instead of the whole sequence. That is lazy evaluation: a chain of generators streams a dataset through several transforms without ever materializing an intermediate list. The tradeoff is that a generator is single-pass and has no length.
What is vectorization in NumPy and why prefer it to a Python loop?
Vectorized operations apply to a whole array at once inside compiled C, so the per-element interpreter overhead and the boxing of Python objects disappear. A loop over a million floats pays a million interpreter steps; the vectorized form pays one dispatch. It also uses contiguous typed memory, which is cache-friendly. The rule in review is that an explicit element-wise loop over a NumPy array is almost always the wrong shape.
What are Python decorators and how would a data engineer use one?
A decorator is a callable that wraps a function and returns a replacement, applied with @-syntax. In pipeline code they carry cross-cutting concerns without touching the body: timing and logging a task, retrying with backoff on a transient failure, caching with functools.lru_cache, or registering a function into a task registry. Use functools.wraps so the wrapped function keeps its name and docstring.
What is a context manager and when do you write one?
An object implementing __enter__ and __exit__, used with the with statement so setup and teardown are guaranteed even when the body raises. You write one when a resource must be released deterministically: a database connection, a file handle, a temp directory, or a transaction that must roll back on error. contextlib.contextmanager turns a generator into one with far less ceremony.
What are dataclasses and why use one over a dict?
A dataclass generates __init__, __repr__, and __eq__ from typed field declarations, giving you a record with named, type-annotated fields. Over a dict it buys attribute access, static checking, and a schema that is visible in the code rather than implied by whatever keys happen to be present. frozen=True makes instances hashable and immutable, which is what you want for a pipeline config.
How does Python manage memory and garbage collection?
CPython uses reference counting as the primary mechanism: an object is freed the moment its count hits zero. Because reference cycles never reach zero, a generational cyclic collector runs periodically to find and break them. The practical consequences are that holding one reference to a big object keeps it all alive, and that __del__ ordering is not something to depend on.
What is the difference between multiprocessing and multithreading here?
Threads share one interpreter and one GIL, so they overlap I/O waits but not CPU work. Processes each get their own interpreter and memory, so they use multiple cores, at the cost of pickling data across the boundary. For a data ingestion task dominated by network waits, threads or asyncio win. For CPU-bound transforms, multiprocessing or an engine that drops into C.
What is pickling and what are its risks?
Pickle serializes a Python object graph to bytes. The risks: unpickling executes arbitrary code, so it must never be pointed at untrusted input; the format is Python-specific and version-fragile, so it is a poor choice for durable or cross-language storage; and it silently drops things like open file handles. Parquet, JSON, or Avro are the right answers for data at rest.
What is the difference between a list and a tuple in pipeline code?
A list is mutable and sized for growth; a tuple is immutable and fixed. In data engineering the immutability is the point: a tuple is hashable, so it can key a dict or join a set, and it signals a fixed-schema record whose shape will not shift under you. Lists are for accumulating; tuples are for records and composite keys.
