Overview
The Alert Microservice (Alert Service) is an independent service in the IoT platform responsible for the full lifecycle management of alerts. It receives alert events from the rule engine, data services, and device services, performs alert aggregation, classification, and notification delivery, and supports closed-loop management including acknowledgment, handling, and closure. In scenarios such as industrial monitoring, energy management, and intelligent operations, if the massive number of abnormal events generated by devices are pushed directly without processing, it will cause an 'alert storm'—operators are overwhelmed by a large number of repetitive, low-value alerts, and the faults that actually require attention may be ignored. The core value of the Alert Microservice lies in 'noise reduction' and 'efficiency improvement.'
IoT alerts differ significantly from IT operations alerts: device numbers can reach millions, alert sources are diverse (offline, threshold exceeded, abnormal fluctuations, communication failures), and notification targets include on-site workers, dispatch centers, and management. Some scenarios require second-level response (such as security and fire protection). The Alert Microservice needs to design high-throughput, low-latency, configurable alert pipelines to address these characteristics.
Core Capabilities
Alert Aggregation and Noise Reduction
- Merge identical alerts: Duplicate alerts from the same device and rule within a time window are merged into one, with the number of occurrences attached
- Rate limiting: Push frequency limits for a single user/single channel to avoid SMS flooding or message spamming on DingTalk
- Silencing Period: Set a silencing period after an alert is recovered, during which the same rule will not be triggered again in the short term
- Alert Aggregation: Aggregate by device group, region, and rule type, supporting summary alerts such as "10 devices offline in a certain region"
Multi-Channel Notification
Supports various notification methods including SMS, email, DingTalk, WeCom, Feishu, app push notifications, and voice calls. Routing can be done based on alert level, duty schedule, and user preferences: urgent alerts via SMS + phone call, general alerts via app push, and batch summaries via email reports. Integrates with third-party notification gateways (Alibaba Cloud SMS, Tencent Cloud SMS, etc.), supporting template configuration, sending records, and retry on failure.
Alarm Escalation and Duty
- Escalation Strategy: Unacknowledged alarms automatically escalate after N minutes, notifying the next level or the duty supervisor
- Duty Rotation: Alarms are assigned to current duty personnel according to the schedule, supporting rotation, random assignment, and load balancing
- Escalation Chain: First Line → Second Line → Third Line, or route to different handling groups based on alarm type
Closed-loop Management
The entire process from the generation to closure of an alert: Trigger → Notification → Acknowledgment (someone takes over) → Handling (fill in handling measures) → Closure . Supports alert statistics, response time analysis, and duplicate alert analysis, providing data support for operation and maintenance optimization. Integrated with the ticketing system and CMDB, it can automatically convert alerts into tickets and associate asset information.
Typical IoT alert types
Device offline alert: Triggered when a device fails to report heartbeat or data for a set period. It is necessary to distinguish between "true offline" (power outage, network disconnection) and "false offline" (device sleep, long batch reporting intervals) to avoid false alarms. Differential timeout thresholds can be set based on device type and historical behavior.
Data anomaly alert: Triggered by sudden changes in sensor data, continuous deviation from the normal range, missing data, etc. For example, temperature and humidity sensor readings suddenly dropping to zero or flow meters showing negative values may indicate device failure or communication issues.
Threshold limit alert: Business threshold alerts such as excessive temperature, pressure overlimit, or insufficient power. Supports multi-level thresholds (warning, alert, critical), with different notification strategies for each level.
Communication and connectivity alert: IoT SIM cards suspended for insufficient balance, weak signal, frequent disconnections, etc., which are particularly important for devices relying on cellular networks. In IoT SIM card management scenarios, SimtoLink often links card status anomalies with alert services, promptly reminding customers to recharge or replace the card.
Coordination with the rule engine
Alert microservices usually do not directly interface with device data but receive alert events output by the rules engine. The rules engine is responsible for "when to alert" (condition judgment), while the alert microservice is responsible for "how to notify" (convergence, routing, pushing). The two are decoupled through a message queue: the rules engine publishes alert events to Kafka/RocketMQ, and the alert service consumes and processes them. This architecture supports dynamic rule adjustments and independent scaling of the alert service.
Technology Selection and Implementation
Open-source solutions like Prometheus Alertmanager and Grafana Loki are suitable for monitoring scenarios; IoT platforms often use self-developed alert modules or ones based on platforms like ThingsBoard or Alibaba Cloud IoT. Key implementation points: event deduplication (based on device + rule + time window key), asynchronous pushing (to avoid blocking the main process), circuit breaking and fallback of notification channels (automatically switching if one channel fails).