Queues & Streaming · lesson 5 of 5 · about 30 min
Nightly jobs versus live windows, and when each is right: how stale a batch answer is, how windows give an endless stream a beginning and an end, why event time and watermarks matter when events arrive late, how checkpoints make stateful jobs recoverable, what reprocessing history costs, and the false spike you get by windowing on the wrong clock. Tap any for a plain-language card.
Two ways to compute
Neither is better in general. Batch is cheaper per record and simpler to reason about; streaming is needed when the value of an answer decays fast: fraud detection, alerting, live dashboards, personalisation. A shorter batch interval shrinks the gap but never reaches zero, and at some point it is a stream with extra steps. How old is the answer for a given schedule?
An hourly batch that takes 10 minutes gives answers that are on average 40 minutes old. A stream gives seconds.
A batch job every 1.0 h that takes 10 min yields answers that are on average 40 min old (1.2 h at worst). A stream processor keeps the answer within seconds of the event. If the answer loses value in minutes, batch is the wrong tool.
Average age = interval ÷ 2 + processing time; worst = interval + processing time. The stream figure is typical, not guaranteed.
Windows
The choice changes the numbers you report: tumbling windows are jumpy at the edges, sliding ones show trends sooner at four times the work, and sessions answer questions (how long was a visit?) that fixed windows cannot. The chart is a seeded stream of about 1,500 events from 40 users over ten minutes.
Switch from tumbling to sliding with the same size: the series gets smoother. Switch to sessions and move the gap to see visits merge and split.
Tumbling windows of 60 s give 10 non-overlapping counts; the busiest has 260.
Real window logic over a seeded stream with a slowly varying rate. Session starts are counted per bucket.
Event time
A is the processor's estimate of “I have seen everything older than this”, typically the largest event time seen minus an allowed lateness. A window closes when the watermark passes its end; events that arrive afterwards are late and are dropped, counted separately, or used to correct the result. A longer allowance drops fewer events but delays every result by that long. This is a pure trade-off between completeness and latency.
With a 2-second typical delay and a heavy tail, allowing 0 seconds drops over a fifth of events; allowing 10 seconds drops about 2% but every result is 10 seconds late.
Allowing 2 s of lateness drops 11.8% of the events as late, and every window result waits 2 s longer than it needs to. That is too many to ignore: allow more time, or route late events to a correction path.
20,000 seeded events, one every 50 ms, with log-normal network delay; 10-second windows close when the watermark (largest event time seen minus the allowance) passes their end.
State and recovery
How often to checkpoint is a trade-off. Each checkpoint pauses or slows the job to save its state; a long interval means less overhead but more input to redo after a crash. The recovery time is the redo plus the restart, and the job must process faster than real time to catch up. Flink uses a variant of the Chandy-Lamport snapshot algorithm to do this without stopping the world.
With 10 GB of state, checkpointing every 10 seconds costs about 100% of the capacity, every 5 minutes about 3%, but recovery grows from seconds to minutes.
Checkpointing 10 GB of state every 1.0 min takes 16.7% of the job's time. After a crash it replays, on average, 30 s of input and needs about 45 s to catch up and restart. Too frequent: the job spends its life saving state.
Pause = state ÷ 1 GB/s; overhead = pause ÷ interval; recovery = (interval ÷ 2) ÷ (catch-up speed − 1) + 30 s restart. Incremental checkpoints reduce the pause in real systems.
Reprocessing
Replaying is bounded by speed. While the job reads history, new events keep arriving, so it must process faster than the live rate, and the time to catch up is backlog ÷ (capacity − arrival rate). A job with only 10% headroom takes ten times as long to catch up as one with twice the capacity, and one with no headroom never catches up. So size reprocessing capacity deliberately (scale out temporarily), and keep the log long enough to cover the history you may need.
A week of 20,000 events a second at 3× capacity catches up in about 3.5 days. At 1.1× it needs 70.
Replaying 7 days (12,096,000,000 events) with 3.0× capacity leaves 40,000 events a second to eat the backlog: 3.5 days to catch up.
Catch-up time = backlog ÷ (capacity − live rate).
Batch or stream?
Batch for big, scheduled, re-runnable computation; streaming when the answer's value decays fast; interactive queries for ad-hoc questions; plain requests for single answers. Each answer is worth 20 XP.
Ask: how fast does the answer lose value? is the input bounded? must it be re-runnable?
Compute the monthly invoices for every customer on the 1st.
Tip: press 1 to 4 to answer, Enter for the next one.
Who lives by this
Four published sources on batch and stream processing.
processing large datasets on commodity clusters
The MapReduce paper described a programming model and runtime that splits a large computation into map and reduce tasks over many machines, re-running tasks that fail. It made reliable batch processing of web-scale data routine and inspired Hadoop.
windows, watermarks and triggers
'The Dataflow Model' paper laid out the vocabulary used by modern stream processors: event time versus processing time, windowing, watermarks that estimate completeness, and triggers that decide when to emit results, with late data handled explicitly. It underlies Apache Beam.
checkpointing with barriers
Flink takes consistent distributed snapshots of operator state by injecting barriers into the stream, a variant of the Chandy-Lamport algorithm, and on failure restores the last snapshot and replays the input from the recorded offsets, which gives exactly-once state updates.
a stream-processing pipeline
Netflix has described Keystone, a data pipeline that moves trillions of events a day from services through Kafka into stream processors and storage, powering analytics and operational insight.
Interview angle
Six questions, each worth 20 XP. Batch versus stream is a common design-interview fork.
Tie the choice to how fast the answer loses value, then talk about windows, event time and state.
How old is a batch answer on average if the job runs every hour and takes 10 minutes?
Tip: press 1 to 3 to answer, Enter for the next one.
Break it
With event-time windows each event is counted in the minute it occurred, so the replay fills in the minutes that were empty and the series looks exactly as it would have without the outage (a late correction to old windows, not a new spike). That is why serious stream processing uses event time with watermarks, and keeps processing time for operational metrics about the pipeline itself. Choose a mode and replay speed.
Five minutes of outage replayed at 6× makes the processing-time series spike to 6,000 a minute against a true 1,000. Switch to event time.
By processing time the chart reads zero for 5 minutes and then spikes to 6,000 a minute while the backlog drains, against a true 1,000. Nothing unusual happened to the traffic.
Traffic is a constant 1,000 events a minute; the processor is down for the outage, then drains the backlog at the chosen multiple of the live rate.
Where to go next