Moving Sensitive Data Safely

Counting the places data is exposed

It arrives as a review: 'walk me through how customer data moves from the checkout service to the analytics warehouse, and tell me where it is exposed.' The word exposed is the hint. The interviewer holds a mental list of places personal data leaks while it moves and is counting how many you find. The list runs longer than encryption: the wire, the staging bucket, the broker's disk, the dead-letter queue, the orchestrator's logs and the error message that printed a row. Past the pipeline itself, it includes the backup and the analyst's laptop after a CSV export. Finding 6 of those 8 earns full marks on the diagnosis.

In transit: stated precisely

Transport encryption with TLS between every hop is assumed, and the credit is for the detail: mutual TLS between internal services so the producer is authenticated too, with certificates issued and rotated by an internal authority. You also get credit for saying that TLS protects the wire only. The moment data lands on a disk, a broker partition, a staging file, a temp table, it is at rest and needs its own encryption, with keys held in a key management service rather than in a config file. Field-level encryption goes further: the sensitive column is encrypted by the producer with a key the pipeline never holds, so a compromised warehouse yields ciphertext.

Masking, hashing, tokenizing: pick by what survives

Most of the tradeoff points in this round come from choosing among 3 techniques, each of which preserves something different. Masking replaces characters, so a phone number becomes *-*-**42: the format and last 2 digits stay readable, no joins are possible, and it's irreversible. Keyed hashing replaces the value with a fixed-length digest computed with 1 secret key shared by every row, usually called a pepper and best applied as HMAC-SHA256 rather than concatenated onto the value: joins across tables still work because equal inputs give equal outputs, and the original cannot be recovered without the key. A random salt per row, the password-storage habit, gives the same email 2 different digests and breaks every join, and a hash with no secret at all falls to a precomputed table in minutes on a low-entropy field like a phone number. Tokenization swaps the value for a random token held in a vault: joins work, the original is recoverable by anyone with vault access, and the vault becomes the thing to protect. The answer that scores starts from which downstream need must survive and works back to the technique: needing the original back means tokenization, joins without that need mean a keyed hash, and masking is left for fields people only have to read.

CREATE VIEW analytics.customers_pseudo AS
SELECT
  to_hex(hmac_sha256(to_utf8(lower(email)), to_utf8('pepper-2024-q4'))) AS customer_key,
  regexp_replace(phone, '\d(?=(?:\D*\d){2})', '*') AS phone_masked,
  signup_date,
  country
FROM raw.customers;

The hash includes a secret pepper, which in production comes from the secret store rather than the view text, and it lower-cases the email first so the same person with mixed-case entries gets 1 key. Point that normalization out in the interview: a pseudonym that splits 1 customer into 2 is a correctness bug dressed as a privacy feature.

Where the masking happens is the scope question

Here the interviewer wants a placement decision. Mask at the source and the raw value never exists downstream of the producer: every exposure point on the earlier list, down to the backups and exports, holds only the masked value. That is the strongest posture, but it blocks any legitimate use of the raw field, such as fraud review. Mask at serving, with a view per audience, keeps the raw column in a restricted table and gives analysts the pseudonymized view: flexible, but every copy of the raw table, including its logs and backups, is now in scope. The defensible answer in most companies is a split: identifiers are pseudonymized at ingest with a keyed hash so analytics joins still work, and the raw value lands in a separate table with its own access policy and a short retention.

Expect 2 follow-ups. Deletion requests against immutable logs are answered with crypto-shredding: encrypt each customer's fields under a per-customer key, and deleting the key renders every copy unreadable without rewriting history. And dead-letter queues, which collect the malformed rows nobody looks at for weeks, are the leak point most teams forget, so you state their encryption and retention explicitly and unprompted.

Do

• Count the leak points: wire, staging, broker disk, DLQ, logs, errors, backups, exports
• Pseudonymize at ingest with a keyed hash so joins survive
• Hold raw identifiers in a separate table with its own policy and retention
• Answer deletion against immutable logs with per-customer keys

Avoid

• Stop at 'we use TLS' when asked where data is exposed
• Hash a phone number without a secret key
• Let a stack trace print a row into the orchestrator's logs
• Forget that the dead-letter queue holds the same fields as the main topic

About this lesson

This section is part of the How Data Moves: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.