The Register: Week 3
The Weekly · Week 3 · Stream
The Register
Next drop Monday Oct 5, 16:00 UTC14 inFirst in sandy_boar_3612
The challenge
You're a data engineer at an electric utility. Each half hour, every smart meter reports the energy it used and its register, the total it has counted since it was installed. A feed forwards each report as 1 JSON line, and billing needs 1 clean row per meter per half hour from it. Write the code that turns the feed into that table.
The mess
The meters came from 3 makers over 12 years, and each maker's system reports in its own way. Meters that lose signal keep their reads and send them when it returns. Every line crosses a message queue on its way to you. A collector that times out sends its whole batch again. Some collectors assemble lines by hand, and a few were cut short.
1,000,000 messages
Click a column for its requirements, or click the bottom of the sheet to see the 10 sample rows to produce.
Notes from billing
- 1 row per meter per 30 minute interval.
- For the same interval, an actual read beats an estimate.
- collector, received_at and other head-end extras are bookkeeping.
- Corrections are copied from the meter database as change events.
How you're scored
- +1the right value. A right null counts.
- 0null where the source held a value. Silence is free.
- −1a wrong value. Silent corruption costs.
- −2a row that should not exist: no matching key, or a second row for the same key. The last row for a key is the one scored.