Pinterest Data Engineer Interview

def related_pins(pin_id: str, top_k: int = 50) -> list[tuple[str, int]]:
    boards = pin_to_boards[pin_id]
    candidates: dict[str, int] = {}
    for board_id in boards:
        for other_pin in board_to_pins[board_id]:
            if other_pin == pin_id:
                continue
            candidates[other_pin] = candidates.get(other_pin, 0) + 1
    # Top-k by co-occurrence count
    return sorted(candidates.items(), key=lambda x: -x[1])[:top_k]

User events -> Kafka (engagement_events topic, key=user_id)
   -> Flink (real-time features, RocksDB state, EXACTLY_ONCE)
        -> Redis (online store, 30-day TTL, p99 < 10ms)
        -> S3 feature log (immutable, event-time partitioned)
   Spark daily batch features:
        S3 events -> Spark -> S3 feature parquet
        -> Iceberg table for query
        -> registered in Galaxy feature catalog

Training data:
   Spark as_of_join between labels (next-day clicks) and feature
   log, joined by (user_id, event_ts) where feature_ts <= label_ts.
   Produces leak-free training data.

Online inference:
   Ranker service reads features from Redis by user_id.
   On Redis miss: fall back to default vectors.
   Drift monitor: daily PSI / KS-test on feature distributions.