Overview
IoT platforms need to execute a large number of scheduled tasks: daily report generation, device offline detection, data archiving, billing settlement, data plan expiration reminders, device health inspections, etc. These tasks are periodic, predictable, and require reliable execution. Distributed task scheduling systems such as XXL-JOB, Elastic-Job, and Quartz Cluster provide task sharding, failover, and visual management, supporting multi-node parallel execution to avoid single points of failure and duplicate execution. Unlike the "continuous running" of stream processing, scheduled tasks are "triggered as planned," suitable for batch processing, reports, inspections, and similar scenarios.
In IoT scenarios, the task volume can be very high: millions of devices need to be inspected in batches by group, and multi-tenants need reports generated separately. Task sharding can split large tasks into multiple sub-tasks executed in parallel by different nodes, significantly reducing the total time. Failover ensures that if one node crashes, its tasks are taken over by other nodes without loss or duplication.
Core Functions
Task Sharding and Parallelism
Sharding can be done by device group, tenant, region, and other dimensions, with each shard executed by different nodes. For example, "offline device inspection" is sharded by tenant: Node 1 executes tasks for Tenant A, and Node 2 executes tasks for Tenant B. Supports dynamic sharding and custom sharding strategies, significantly improving the execution efficiency of large-scale tasks.
Failover and Cron
When a node crashes, its incomplete or pending tasks are automatically migrated to other nodes. Supports task dependencies and blocking strategies. Cron expressions can be flexibly configured for execution cycles (e.g., 0:00 daily, every hour, every Monday). Supports one-time tasks and delayed tasks.
Execution logs and alerts
Task execution history, logs, duration, and status. Failure alerts, retry strategies, manual intervention. Visual management interface, task start/stop, parameter configuration, execution monitoring.
Typical IoT Applications
Daily, weekly, and monthly report generation. Offline device inspection (periodically scanning the online table, marking as offline if no heartbeat within timeout). Data archiving (migrating cold data to object storage). Data plan expiration reminders, billing settlement. Deep integration with IoT business, serving as a supporting component for platform operation and maintenance as well as operations.
Who owns a scheduled trigger
The scheduler manages the right to fire a task instance
Across nodes, a scheduler decides which clock has reached the trigger, which node receives the right to execute, and where that right moves after a failure. How a report is totaled, and how offline is judged, still live in the task code. What the scheduler knows is the task identifier, the trigger expression, the shard parameters, the blocking policy, and the execution record. One trigger creates one or more instances. An instance must end as success, failure, or timeout. Whether the process is still present is not a way to guess the outcome.
A stream job that runs continuously has no moment when "this run ended." The scheduler is for batch work with a start and an end: a daily report, an archive, an expiry scan, a reconciliation. Putting second-level alerts into the scheduler so it polls a database turns the trigger interval into alert delay, and the database is scanned full again and again. Leave second-level reaction on the event path. The scheduler handles work that can wait until the next cycle to finish.
Batch work that can be split into parallel shards
A task suits sharding when the data can be cut by tenant, region, or a range of device identifiers, and the shards do not need to see one another's intermediate results in the same run. An inspection can be sharded by tenant, and each shard scans only its own online records. An archive can be sharded by date and prefix. When a summary must see every device before it can produce one table, shards can only produce partial sums, and one summary instance merges them at the end. Each shard must not write that final report.
For a shard to move after a node fails, the task must also be safe to enter again: running the same shard twice gives the same result as running it once, or the second run finds that the first already finished and exits. Without that property, a move causes a second deduction, a second notice, or a second archive. Parameters should carry the business date and the shard bounds. Do not depend on a default such as "today" that changes with the moment of execution. Otherwise a catch-up of yesterday's task rewrites today's data.
Missed triggers, overlapping runs, and blocking policy
When the scheduler node is down across a trigger point, decide in advance whether to catch up, collapse the misses into one run, or wait for the next cycle. Catching up every missed hour after the node returns produces dozens of reports in a row. Skipping the catch-up entirely leaves that day's close missing. Write the policy on the task. Do not hide behind a global default for the whole cluster. When node clocks disagree, two nodes may both believe they should fire. The right to execute can come only from an election or a database lock.
A long task that is still running when the next trigger fires is the usual case. The blocking policy must say whether to drop the new run, queue it, or stop the previous run. If stopping means killing a process that is writing a file, a half-written object is left behind. If the queue has no length limit, recovery piles up a long string of catch-up runs. An execution record should show which firing this was, which shard it received, and which keys it wrote. Failure events can be handed to the Alert Microservice. There is no need for every task to carry its own notification. The device scope should read groups from the Device Management System, so the schedule parameters do not copy a device list that has already expired.
Shard bounds should use a stable range of identifiers, not a row number that changes as pages change. After devices are deleted or added, two shards cut by page number overlap or leave a gap: one device is inspected twice, and another is never scanned. Write the range into the task parameters, and keep it with the execution record. A manual catch-up should reuse the range from that time. Do not cut the pages again from the device count on the day of the catch-up.