The Behavioral Round

Situation: At my last company we ran a daily ETL on customer events that took 7 hours and frequently missed the 06:00 SLA. The 4 downstream BI dashboards were stale 30% of mornings.

Task: I was asked to bring the SLA to 100% reliability without growing the cluster.

Action: I profiled the job and found 80% of runtime was a single join with severe skew on a customer_id hot key. I rewrote that stage with salting and pre-aggregation, replaced a full reprocess with an incremental merge, and added a backfill DAG for the rare days the job failed.

Result: Runtime dropped from 7 hours to 90 minutes. SLA compliance went to 100% over the next 6 months. Cluster cost dropped 18% because the job now needs less peak compute.

Decision: Looking back, I would have proposed the salting fix before the SLA crisis, not after. I had noticed the skew in a profiling pass two months earlier and deprioritized it. The lesson is that performance work compounds, and waiting for a crisis costs the team trust.

Situation: A VP of Analytics requested a real-time dashboard for a metric that we computed via daily batch. Their stated reason was “we need real-time visibility into revenue.”

Task: Decide whether to build the real-time pipeline (8 weeks of work and ~$60K/year in streaming infrastructure) or push back.

Action: I interviewed three of their analysts about how they would actually use the dashboard. Two said they checked it once a day. One said they wanted “intra-day” visibility but specifically meant 2 to 4 hour latency. I proposed a 2-hour micro-batch as a compromise: 1 week of work, no streaming infra. I wrote a one-pager with the cost comparison and the interview findings, and shared it with the VP before our next meeting.

Result: The VP agreed to the 2-hour micro-batch. We shipped in 6 days. Six months later, the team had not asked for sub-2-hour latency once.

Decision: Looking back, I should have done the user research before the kickoff meeting, not after. The VP felt blindsided by the pushback in the meeting. Sharing the one-pager async a day in advance would have made the same outcome feel collaborative instead of confrontational.

Situation: I led a migration from a legacy ETL framework to Airflow. Three months in, we cut over a critical billing pipeline.

Task: Ensure zero data loss during the cutover.

Action: I set up the Airflow DAG, tested in staging, and ran a side-by-side comparison for two weeks. The numbers matched, so I cut over.

Result: Two days later, the finance team flagged that monthly revenue was off by $40K. Root cause: the Airflow DAG used UTC timestamps, the legacy job used Pacific time. The side-by-side comparison ran for 14 days, and the issue only manifested at month boundaries, which my comparison window had skipped.

Decision: The fix was a one-line timezone correction. The lesson was that comparison windows must include the boundary conditions of the system under test. Now I always test cutovers across at least one full business cycle (month, quarter, fiscal year-end), not just calendar days. I also added a one-page pre-cutover checklist for the team that explicitly lists timezone, locale, and currency as side-by-side validation requirements.