Tiered Storage

Concepts covered: Log storage and retention

A partition that keeps 90 days at 2 TB a day needs 180 TB of broker disk, and broker disk is the costly kind: fast, local and sized for every partition the broker holds. Most of those bytes are read once, by the readers who keep up, and then sit until retention deletes them. Tiered storage moves closed segments off the broker to an object store, a system that holds files cheaply and serves them over the network at some latency, while the topic keeps reading as 1 log. A reader asking for offset 0 still gets offset 0; the broker fetches the segment from the store and hands the records over.

A local retention and a total retention

Enabling it adds a second retention to the topic. retention.ms and retention.bytes keep their meaning: how much the topic holds in total, local and remote together. local.retention.ms and local.retention.bytes say how much of that stays on the broker's disk; their default, -2, means the same as the total, which is the same as not tiering at all. A topic with retention.ms of 90 days and local.retention.ms of 1 day holds 1 day of segments on fast disk and all 90 days in the store. Reads inside the last day are served from local files; reads older than that wait for a fetch from the store.

remote.storage.enable=true
retention.ms=7776000000
local.retention.ms=86400000

A segment's life

Step 3 happens shortly after the roll, and it is the step everything waits on. The local copy is deleted at step 4 only if step 3 completed, so a store that is unreachable for an hour means an hour of segments piling up on the broker past local.retention.ms, which is the right failure: data is never dropped to meet a local target. The active segment is never copied, whatever its age, because it is still being written. Each remote segment is known to the broker by a record in the internal topic __remote_log_metadata, saying which segment lives where and which offsets it covers, so a broker that restarts rebuilds its view of the store from that topic rather than listing the store.

Reading from the store

A fetch that asks for an offset older than the local log's first offset becomes a remote read. The broker looks up the segment in its metadata, finds the position inside the segment using the remote copy of the offset index, which it caches locally under remote.log.index.file.cache.total.size.bytes, downloads the needed part of the segment, and returns records from it. The first such read on a cold segment takes the store's latency, tens to hundreds of milliseconds, in place of a local disk read; remote.fetch.max.wait.ms caps how long the broker waits on the store before answering the fetch with what it has. A lookup by time picks the remote segment by the largest timestamps in __remote_log_metadata, then waits on the store for its time index and records. A reader replaying from 60 days ago runs at the store's throughput, and a reader 10 minutes behind never touches it.

Turning it off starts with remote.log.copy.disable=true, which stops new copies while remote reads keep working, and a topic can stay that way while its remote data is needed. remote.storage.enable=false then makes it local, and the broker accepts that only alongside remote.log.delete.on.disable=true, which deletes the remote segments, so a reader positioned in one must be moved first.

Do

• Size local.retention.ms to cover the readers who replay routinely
• Monitor local disk against local.retention.ms; a gap means copies are failing
• Test a 60 day replay once to learn the store's throughput

Avoid

• Tier a topic whose readers routinely rewind past the local window
• Treat retention.ms as free once bytes are remote
• Disable remote.storage.enable while a reader is positioned in a remote segment

About this lesson

This section is part of the The Log: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.