Overview
Stream processing performs real-time computation on data streams reported by IoT devices without waiting for the data to be stored for batch processing. Device data is continuously generated in the form of a "stream," and the stream processing engine computes it as it arrives, enabling alerts, aggregation, and analysis with second-level or even millisecond-level latency. Compared with rule engines, stream processing supports more complex computation logic: multi-stream correlation, window aggregation, CEP complex event detection, and stateful computation. Flink, Spark Streaming, and Kafka Streams are mainstream choices, with Flink excelling in low latency, exactly-once, and state management, making it widely used in real-time IoT scenarios.
Typical architecture for IoT stream processing: device data is ingested via MQTT/Kafka, Flink jobs consume Kafka topics to perform filtering, aggregation, correlation, and alert evaluation, and results are written to time-series databases, alert services, or sent back to devices. It complements rule engines: rule engines are suitable for simple "condition-action" tasks, while stream processing suits complex computations, multi-stream correlation, and state accumulation.
Core Capabilities
Window computation
Tumbling windows, sliding windows, session windows. Windows are defined by time or count, performing SUM, AVG, COUNT, and other aggregations within the window. Supports event time, processing time, and watermark mechanisms to handle out-of-order events. In IoT scenarios, 1-minute and 5-minute tumbling windows are commonly used for real-time aggregation.
CEP and state management
CEP (Complex Event Processing): Detect event sequence patterns, such as "B occurs within 5 minutes after A." Stateful computation: state accumulation across messages (e.g., moving average, cumulative count). Exactly-once semantics: no duplication or loss after failure recovery. Flink's checkpoint mechanism ensures state consistency.
Multi-source and sink
Supports Kafka, MQTT, HTTP, databases, etc., as data sources and outputs. Can interface with time-series databases, alert services, message queues, APIs, etc. Supports dynamic table addition and schema evolution.
Typical IoT Applications
Real-time alerts: temperature over-limit, device offline, anomaly detection. Real-time aggregation: usage summary by device, region, time window to support dashboards. Anomaly detection: real-time anomaly identification based on statistics or ML models. Data synchronization: multi-source data integration, format conversion, writing to multiple targets. Works with rules engines, data microservices, and alert services to build a complete real-time data processing chain.
How stream state lines up with event time
How events that keep arriving are placed in windows
A stream job keeps computation state while events are still arriving. It accumulates by key, summarizes by window, recognizes a group of events in order, and writes the result downstream. The input is an ordered or nearly ordered stream of events. State hangs on a key, for example one device's count in the latest window. The job does not assume those points already lie in a table that can be scanned over and over. When the job restarts, a checkpoint restores the state together with the read position on the stream, and computation continues from the last position that was consistent.
Tumbling windows meet end to end and suit minute totals that do not overlap. Sliding windows recompute on a step and suit watching whether a recent period crossed a limit. Session windows are cut by an idle gap and suit gathering a burst of reports into one stretch. Whether a window uses event time or processing time decides whether the result follows the device clock. Field questions usually ask when something happened on the device, so carry the device time, and also keep the ingest time for checking.
When a reporting rhythm makes watermarks worth enabling
A watermark means that data earlier than this event time has, given the disorder seen so far, probably all arrived. It suits a stream whose reporting interval is relatively stable and whose disorder has an upper bound. When devices often stay offline for hours and then backfill, the watermark is either held back so windows refuse to close, or the backfill is judged late and dropped. That kind of stream should put the backfill in its own partition, or allow a closed window to be updated under a revision rule, and it should not be cut hard by a very tight watermark.
Before enabling a watermark, measure how disorder is distributed: most points a few seconds late, or a batch of points hours late. Set the slack from that distribution, and send later points to a side output. The stream job does the computation. Real-time Analysis reads results that have already been published. Window definitions and grouping keys on both sides must match, or looking the stream figures up again will disagree.
Out-of-order clocks, backpressure, and end-to-end delivery
If device time is set into the future, the watermark jumps with it, and normal points that arrive afterward are all treated as late. The job should limit how far a single point can push the watermark, and an abnormal clock goes to the side output. When one key receives far more data per second than the others, the parallelism that holds that key becomes the bottleneck and overall latency rises. Split an overheated key, or rate-limit and sample at the access layer. Adding partitions that have nothing to do with that key does not remove the hotspot.
Backpressure, when the downstream cannot keep up with writes, is a signal that the rates do not match. Growing the in-memory buffer further drags the job until checkpoints time out. If the upstream is Message Queue Technology, the consume position must be committed together with the checkpoint, or recovery reads again from a position already processed. Exactly-once means the job state and a downstream that supports transactions commit together. If the downstream only supports an ordinary insert, recovery can still write one extra row. The business table that receives the result needs an idempotency key. If pattern detection writes "appeared one after another within a few minutes" in processing time, network jitter manufactures a false sequence.
The key decides where state lands. Using the device identifier as the key makes window statistics follow the device. To use region as the key, finish the mapping from device to region before the event enters the window. A mapping table changed afterward does not go back and rewrite windows that have already closed. After parallelism changes, state is laid out again on the new partitions, and the old partitions in the checkpoint must still match on recovery. Changing the key algorithm while windows are still open writes the recovered counts onto another set of keys, and usage suddenly halves or doubles. Before changing the key, let the old windows close, and compare the old and new results over an overlapping stretch of time.