Overview
Data Microservices (Data Service) are independent services in IoT platforms responsible for the full-process handling of data reported by devices, covering capabilities such as real-time access, time-series storage, multidimensional queries, aggregated statistics, and data export. Unlike Device Microservices (which manage the device lifecycle), Data Microservices focus on "data" itself: how to efficiently receive, store, and retrieve massive amounts of time-series data. IoT sensor data features high write rates, strong time-dependence, and multiple dimensions (device × metric × time), which traditional relational databases struggle to handle. Data Microservices typically use time-series databases like InfluxDB, TDengine, TimescaleDB, or combined solutions such as Kafka + ClickHouse to support scenarios like monitoring dashboards, report analysis, AI training, and real-time alerts.
Architecturally, Data Microservices usually act as consumers of message queues, pulling raw messages reported by devices from Kafka/RocketMQ, parsing them, and then writing them into time-series databases according to the data model, device ID, and timestamp. They also provide RESTful or GraphQL query APIs externally for frontend applications, rule engines, and reporting systems to call. The data flow is "Device → Access Gateway → Message Queue → Data Microservice → Time-Series Database," and the query flow is "Application → Data Microservice API → Time-Series Database."
Core Capabilities
Data Access
Consume device-reported messages from the message queue, parse JSON/binary formats, and verify the validity of device IDs, timestamps, and metrics. Support batch writing and backpressure control to prevent message queue backlog. Can connect to multiple topics (e.g., by product, by device type) to achieve data separation and isolation. Integrated with the data cleaning module to filter outliers and fill in missing values before storage.
Time Series Storage
Efficiently store data by device, metric, and time. The time-series database uses columnar storage, high compression, and time partitioning, supporting million-level TPS writes and millisecond-level queries. Data retention policies can be configured: hot data retained for 7 days, warm data for 30 days, and cold data archived to object storage. Supports multi-tenant isolation, partitioned by tenant ID or product ID.
Query API
Time range queries, multi-metric aggregation, downsampling (e.g., aggregate minute-level data into hourly or daily). Supports filtering by device, device group, product, region, and other dimensions. API design should consider pagination and streaming return for large data volumes to avoid OOM from fetching too much data at once. Can provide SQL or SQL-like interfaces to lower the usage threshold.
Data Export
Supports exporting in CSV, Excel, and JSON formats for reports, audits, and offline analysis. For large-scale export, use asynchronous tasks + file download links to avoid long-term blocking. Work in conjunction with data permissions and masking policies to ensure exported data complies with regulations.
Typical IoT Applications
Energy Monitoring: Electricity, water, and gas meter consumption data, supporting aggregation by household, building, or area, used for energy consumption analysis, peak-valley statistics, and abnormal electricity usage detection.
Environmental Monitoring: Sensor data such as temperature, humidity, PM2.5, and air quality, supporting environmental monitoring and alerts in smart campuses, smart agriculture, and smart buildings.
Equipment Operating Data: Operational parameters of industrial equipment such as vibration, temperature, and current, used for predictive maintenance, fault diagnosis, and energy efficiency analysis.
Production Statistics: Production line counts, output, yield rate, supporting production reports, capacity analysis, and OEE calculation.
Collaboration with Rule Engine and Stream Computing
Data microservices are responsible for 'storing' and 'retrieving', while the rule engine handles 'calculation' and 'alerts'. After equipment data is written to the time series database, the rule engine can subscribe or query it periodically for threshold checks and complex event processing. Stream computing (such as Flink) can directly consume message queues for real-time aggregation and window calculations, with results written to the time series database or pushed as alerts. Data microservices, the rule engine, and stream computing form a complete 'storage—computation—alert' data chain.