Overview
Message Middleware is the data hub of IoT platforms, responsible for receiving, storing, and distributing data reported by devices. In high-frequency data scenarios such as connected vehicles, industrial internet, and energy monitoring, massive numbers of devices generate hundreds of thousands or even millions of messages per second. Message middleware provides high throughput, low latency, and persistence capabilities to support various consumers including real-time analysis, offline storage, rule engines, and alert services. Unlike point-to-point RPC, message middleware uses a publish-subscribe model, decoupling producers and consumers. New consumers do not require changes to producers, making the system more scalable.
IoT message middleware is usually divided into two layers: ①Device Access Layer: MQTT Brokers (such as EMQX, Mosquitto) handle device connections and message transmission, using the Topic mechanism; ②Data Stream Layer: Kafka, RocketMQ, etc., handle message persistence, multi-consumer distribution, and replay. The two are connected through bridging or rule engines, forming a "Device → MQTT → Kafka → Multi-Consumers" pipeline.
Core Functions
Publish-Subscribe and Topic
Topic Mechanism: Devices publish to iot/device/{deviceId}/data, while the rule engine and data services subscribe to iot/device/+/data. Wildcards (+, #) are supported. Flexible routing meets the subscription needs of different consumers. Both MQTT and Kafka use a similar model.
Message Persistence and Replay
Messages are persisted to disk, and the retention period can be configured. Consumption can be replayed by offset or time, facilitating new consumers to fetch historical data, fault recovery, and audit analysis. IoT data requires both timeliness and replayability.
Horizontal Scalability and Multi-Protocol Support
Partitions, replicas, and consumer group mechanisms support horizontal scalability. Protocols such as MQTT, AMQP, and Kafka Protocol are supported to meet different communication needs among devices, applications, and systems. In IoT scenarios, the EMQX (device access) + Kafka (data stream) combination is commonly used.
Typical Products and Selection
MQTT Broker: EMQX, Mosquitto, HiveMQ. Responsible for device access, lightweight, supports QoS.
Big Data Stream: Kafka, RocketMQ, Pulsar. High throughput, persistent, multiple consumers.
Enterprise Messaging: RabbitMQ. Feature-rich, flexible routing, suitable for complex business logic.
IoT scenarios recommend EMQX + Kafka: EMQX receives device messages, the rule engine bridges to Kafka, downstream data services, real-time analytics, alerts, etc., consume from Kafka. See Distributed Message Queue, Message Queue Technology.