Overview
Data cleansing is a key process in IoT data handling, used to address issues like abnormal values, missing values, duplicate data, and inconsistent formats reported by devices, improving data quality and providing reliable input for subsequent analysis, alerts, and AI training. IoT devices are affected by environmental interference, sensor drift, communication failures, firmware defects, and other factors, so the data they report often contains noise. If this data is directly stored or used in computations, it may lead to false alerts, distorted reports, and biased model predictions. Data cleansing involves validation, filtering, completion, and standardization before data storage (real-time streams) or after storage (offline batch processing) and serves as the infrastructure for data governance.
The challenges of IoT data cleaning lie in the large data volume, high real-time requirements, and diverse device types that make it difficult to unify rules. The reasonable ranges for temperature sensors and humidity sensors differ, and the "normal" fluctuation of the same device varies across seasons. Cleaning rules need to support flexible configuration by device type, product, and scenario, and can combine historical statistics and machine learning for adaptive anomaly detection.
Core Functions
Anomaly Detection
- Range Check: Values that exceed physical or business reasonable ranges (e.g., temperature -100°C, humidity >100%) are directly discarded or flagged.
- Sudden Change Detection: If the value change between adjacent time points exceeds a threshold (e.g., temperature rises 10°C in 1 second), it may indicate sensor malfunction or communication error.
- Outlier Detection: Identify abnormal values using statistical methods (e.g., 3σ, IQR) or machine learning, suitable for scenarios without obvious physical thresholds.
- Logical Consistency: For example, a flow meter reading should not be negative, and cumulative values should be monotonically increasing.
Missing Data Handling
Data loss may occur due to device network disconnection, reporting failure, or uneven sampling intervals. Handling strategies include: Linear Interpolation (linear fill between adjacent points), Forward Fill (use the previous value, suitable for slowly changing quantities), Discard (discard the record if a key indicator is missing), Mark as Missing (retain timestamp for identification in subsequent analysis). The choice of strategy should be combined with the business context: report statistics may allow interpolation, while alert rules usually require real data.
Remove duplicates
Repeated reporting by devices, repeated consumption of message queues, and multi-path collection lead to duplicate data. Deduplication is based on the unique key of device ID + timestamp + indicator, and the first or latest one is retained. In high-throughput scenarios, Bloom filter + Redis can be used to achieve rapid deduplication.
Format conversion and standardization
Units should be unified (for example, temperature should be unified to degrees Celsius, length should be unified to meters), timestamps should be unified to ISO 8601 or Unix milliseconds, and enumeration values should be standardized (for example, "on"/"ON"/"1" should be unified into "on"). Aligned with the object model to ensure that data from different devices and manufacturers can be uniformly parsed and aggregated.
Typical IoT Scenarios
Sensor Drift Correction: Temperature and humidity sensors may drift over time and require periodic calibration or correction using reference values. Automatic correction can be achieved by combining historical data and cross-validation with multiple sensors.
Data Completion: Reports and dashboards require continuous time series data, so missing points need to be filled. In batch processing scenarios, historical data can be completed offline.
Report Data Quality: Energy reports, production statistics, etc., need to remove outliers to prevent a single anomaly from inflating or deflating overall metrics. Rules for replacing outliers can be set (e.g., using the mean of preceding and following values).
AI Training Data: Machine learning models are sensitive to dirty data, so the data must be systematically cleaned before training. Missing or abnormal data can lead to model overfitting or underfitting; the quality of cleaning directly affects model performance.
Implementation Methods
Data cleaning can be integrated into the rules engine (adding validation logic in real-time rules), stream processing (adding cleaning operators in Flink/Spark Streaming), standalone cleaning services (consume from Kafka, clean, and then write). In real-time scenarios, stream processing is preferred; in offline scenarios, batch processing is prioritized. In IoT data management practice, SimtoLink recommends adding a cleaning step between the data access layer and the storage layer to ensure controllable quality of the stored data.