Streaming architecture doesn't have to mean eye-watering infrastructure bills. We walk through how Apache Flink enables resilient, low-latency pipelines at scale — and the architectural choices that keep costs from spiraling out of control.
The first question to answer before adopting stream processing is whether you need it, and the honest answer for a lot of teams is no. A batch job that runs every fifteen minutes is dramatically simpler to build, operate, debug and reason about than a streaming pipeline, and for a large share of analytics workloads fifteen minutes is indistinguishable from real time as far as the business is concerned.
Streaming earns its complexity when latency is part of the product — fraud decisions, live inventory, operational alerting, anything where a fifteen-minute-old answer is not merely stale but wrong. If that describes your use case, Flink is a strong choice, and the cost concerns that surround it are mostly a consequence of a handful of avoidable design decisions.
Why Flink rather than the alternatives
Flink's distinguishing feature is that it treats state as a first-class concern. Stateful operators, exactly-once semantics via checkpointing, and proper event-time processing with watermarks are built in rather than bolted on. For anything involving windowed aggregation over out-of-order data — which is to say, most real event streams — this matters enormously.
The trade-off is that Flink expects you to understand its state model. Teams that treat it as a black box end up with pipelines that work in testing and fall over in production the first time a checkpoint takes longer than the interval between checkpoints.
Event time versus processing time
This is the concept that causes the most production incidents, and it is worth being precise about. Processing time is when your job saw the event. Event time is when the event actually happened. They differ because networks are slow, mobile clients buffer while offline, and upstream systems retry.
If you window by processing time, a mobile client that reconnects after two hours offline will have its events counted into the current window. Your hourly totals will be wrong in a way that is difficult to detect and impossible to correct after the fact. Watermarks exist to solve this: they let the job reason about how late data can be before a window is closed.
Choosing processing time because watermarks seem complicated is choosing to be quietly wrong instead of visibly complicated.
Where the money actually goes
Streaming infrastructure bills are usually dominated by three things, and only one of them is compute.
State size
Large keyed state means large checkpoints, slow recovery and expensive storage. The most common cause of unbounded state growth is a keyspace that grows without limit — keying by session ID or request ID without a state time-to-live. Set TTLs on keyed state deliberately rather than discovering the need during an outage.
Checkpoint frequency
Checkpointing every ten seconds when your recovery objective is five minutes is pure waste, paid continuously. Set the interval from the actual recovery requirement, use incremental checkpoints with RocksDB for large state, and monitor checkpoint duration as a leading indicator of trouble.
Over-provisioned parallelism
Teams frequently set parallelism high 'to be safe' and then run at ten percent utilisation permanently. Parallelism should follow the partition count of the source and the measured throughput per task, not intuition.
Operational habits that prevent incidents
- Monitor consumer lag per partition, not in aggregate. One stuck partition hides completely in an average.
- Alert on checkpoint duration trending upward. It is the earliest visible symptom of state growth becoming a problem.
- Practise restoring from a savepoint before you need to. Recovery paths that have never been exercised do not work.
- Version your pipeline's state schema explicitly, because you will need to change it while the job is running.
- Keep a replayable source. If you can rebuild the output by reprocessing the input, most bugs become recoverable rather than permanent.
The pragmatic recommendation
Start with the smallest streaming component that delivers the latency the business actually needs, and leave everything else in batch. Hybrid architectures are not an admission of failure — they are usually the correct answer, because they confine operational complexity to the part of the system that genuinely requires it.