Overview
Distributed message queues are the core infrastructure for IoT platforms to handle massive amounts of device data, serving as a "data bus." Data reported by devices through access gateways is published to topics in message queues like Kafka or RocketMQ via the gateways. Rule engines, data microservices, real-time analytics, log systems, and other consumers independently subscribe and retrieve what they need. This architecture decouples device data from downstream systems: adding a new consumer (such as a new alarm rule) does not require changes to the data producers; consumers can scale independently; message persistence supports replay and repeated consumption, facilitating fault recovery and data reprocessing.
In IoT scenarios, the message volume can reach hundreds of thousands or even millions per second. A single-node MQ cannot support this load, necessitating a distributed message queue. Kafka, RocketMQ, Pulsar, and others achieve horizontal scaling and high availability through partition, replica, and consumer group mechanisms. Topic design is usually partitioned by device type, product, or tenant, facilitating isolation and scaling.
Core Features
High Throughput and Partitioning
Partitions allow parallel writes; single partitions write sequentially to disk, while multiple partitions enable parallelism across machines. A single cluster can support millions of messages per second. IoT device data is hashed by device ID or product ID to different partitions, ensuring ordered data for the same device while allowing parallel processing for different devices.
Persistence and Replay
Messages are persisted on disk with configurable retention periods (e.g., 7 days). Offset-based replay consumption is supported, enabling new consumers to catch up on historical data or replay after failures. IoT data sometimes requires post-analysis and auditing, making persistence essential.
Consumer Groups and Exactly-once
Load balancing among multiple consumers in a consumer group, with a single partition consumed by only one consumer in the group. Supports horizontal scaling. Transactional messages and idempotent writes can achieve exactly-once semantics, ensuring no data is duplicated or lost. Scenarios like IoT data storage and alarm triggering have high consistency requirements.
Typical IoT Applications
Device Data Collection: The access gateway publishes the device-reported data to the iot-data Topic, which is consumed by the data microservice for writing to the time-series database, and by the rules engine for threshold evaluation.
Log Aggregation: Device logs and platform logs are uniformly written to the log Topic for consumption by the logging system and audit system.
Event-Driven: Events such as device online/offline and alarms are published to the event Topic, triggering downstream actions like workflows, notifications, and work orders.
Data Synchronization: Data synchronization across regions and tenants, delivered asynchronously via message queues.
Topic Design Recommendations
Divide by business domain: iot-data (device data), iot-events (device events), iot-alerts (alarms). Creating Topics by tenant or product can achieve multi-tenant isolation. The number of partitions is recommended to be an integer multiple of the number of consumers to ensure load balancing. The message body should use JSON or Protobuf and include device ID, timestamp, data, metadata, etc.