Logical vs Physical Device Identity: Surviving IoT Hardware Swaps

IoT device lifecycle management breaks when history follows the hardware. Why I skipped the obvious cleanup, which would have misrouted about 900 live devices.

In IoT device lifecycle management, customers think of a device as a spot on their map, while the circuit board bolted to it can be swapped out. When history follows the board instead of the spot, routine hardware swaps can split a location's history in two, and the cleanup that looks obvious can do more damage than the original bug.

On a recent engagement, a roadside sensor fleet's histories were splitting whenever a unit was swapped through the wrong path. The fix on the table was a quick cleanup of over a thousand odd-looking records. I checked both databases before running it, and it would have misrouted lookups for about 900 live devices and shown one customer another customer's images. So I didn't run it. I built a lifecycle model that keeps one history per location through every swap, and made the wrong paths hard to take.

Where the histories were splitting

The fleet was a few thousand roadside sensor stations. Every reading and image was stored against a logical device ID, not the physical unit's hardware ID. Units fail, get swapped, get refurbished and get redeployed, often to a different customer. To the customer, "the same device" means the same spot on their map.

My 2023 design for this fleet had a proper replace operation that swapped the unit and kept the history. But the tool people actually reached for was a "retire" shortcut, and when someone used it for a replacement it minted a new device ID for the same location, splitting the location's history in two. The fault was in the tooling: the right path was harder to find than the wrong one.

The audit turned up more:

• Over a thousand records carried a marker prefix on their hardware ID, almost all in a state that shouldn't exist.
• The customer-facing replacement flow had been built twice and switched on for no one.
• State transitions weren't enforced, the swap was atomic only because of a database driver default, and two records could hold the same unit.

An audit log of every device change

In 2023 I also added a lineage table to the fleet's device database. Three triggers on the device table write a row for every change, with the old value, the new value and a timestamp:

• Setup. A row when a device is first created, with its starting state.
• Activation state. A row for every state change, from provisioned to activated to retired.
• Customer account and unit. A row whenever a device moves to another customer's account, which is how a unit gets dispatched to a customer or comes back from the field, and whenever its hardware ID changes.

The triggers run inside the database, so every writer is covered, from admin tools to one-off scripts, and none of them has to remember to log.

CREATE TRIGGER trg_device_lineage ON devices AFTER UPDATE AS
INSERT INTO device_lineage (device_id, old_state, new_state, old_account_id, new_account_id,
                            old_hardware_id, new_hardware_id, changed_at)
SELECT d.id, d.state, i.state, d.account_id, i.account_id, d.hardware_id, i.hardware_id, SYSUTCDATETIME()
FROM inserted i JOIN deleted d ON d.id = i.id
WHERE d.state       IS DISTINCT FROM i.state
   OR d.account_id  IS DISTINCT FROM i.account_id
   OR d.hardware_id IS DISTINCT FROM i.hardware_id;

CREATE TRIGGER trg_device_lineage ON devices AFTER UPDATE AS
INSERT INTO device_lineage (device_id, old_state, new_state, old_account_id, new_account_id,
                            old_hardware_id, new_hardware_id, changed_at)
SELECT d.id, d.state, i.state, d.account_id, i.account_id, d.hardware_id, i.hardware_id, SYSUTCDATETIME()
FROM inserted i JOIN deleted d ON d.id = i.id
WHERE d.state       IS DISTINCT FROM i.state
   OR d.account_id  IS DISTINCT FROM i.account_id
   OR d.hardware_id IS DISTINCT FROM i.hardware_id;

For almost three years the table was written and rarely read. This year it paid off:

• It made the diagnosis possible. The retire-then-recreate pattern, the units reused across customers and the illegal state transitions all came out of its rows.
• It made the relabel safe. Before the migration cleared a retired record's hardware ID, it checked that the lineage held that record's full history.
• It got a reader. The client's backend lead built a change history for each device in the admin portal on top of it.

Why the obvious cleanup wasn't safe

The first plan was to strip the prefix from those records and dedupe them. I wasn't convinced I had the whole picture, so I checked both the system of record and the analytics store first.

Nearly all of the prefixed records had a live successor holding the clean hardware ID, and both lookup paths took the first match: one returned the oldest row, the other an arbitrary one. Dashboards filter by customer account only, so leaving retired records in the account is how customers keep read-only access to images from decommissioned units. And nine in ten of the prefixed records had a twin only in a different customer's account, so merging histories by hardware ID would have put one customer's images on another customer's dashboard. Only a few dozen pairs were genuinely safe to combine. Every odd-looking field was doing a job, and I closed the cleanup ticket without running it.

A reusable IoT device lifecycle model

Two identities. The logical device is a location with one continuous history. The physical unit is a circuit board that can belong to several locations over its life. The device record holds the current binding, and the history table holds the lineage.

A five-state lifecycle in one shared library. Every service now goes through the same operations, checked against a table of legal transitions:

The swap had to be a single statement. SQL Server checks a unique index after each statement, not at commit, so a two-step swap would collide with its own uniqueness rule halfway through. A review of the plan caught that the obvious unique index would have broken Replace outright.

-- Swap units between a location's record and a pool record in one statement,
-- so the unique index on hardware_id never sees two rows holding the same unit.
UPDATE devices
SET hardware_id = CASE id WHEN @location_id THEN @new_unit
                          WHEN @pool_id     THEN @old_unit END
WHERE id IN (@location_id, @pool_id);

-- Swap units between a location's record and a pool record in one statement,
-- so the unique index on hardware_id never sees two rows holding the same unit.
UPDATE devices
SET hardware_id = CASE id WHEN @location_id THEN @new_unit
                          WHEN @pool_id     THEN @old_unit END
WHERE id IN (@location_id, @pool_id);

The rollout order mattered. Shared code was vendored into many services, and the analytics copy lagged behind the system of record.

A required choice at setup. Installers now have to say whether a unit is going to a new location or replacing an existing one. There's no default, and if the list of existing locations fails to load, setup stops rather than guessing "new". I rejected registering replacements at shipping time, because nobody kept those flags current, and I rejected telemetry heuristics, because dying units don't reliably go quiet.

The client's backend lead reviewed the shared library and added the rule that hardware attributes travel with the unit on a swap.

Results

One site that an old shortcut had split was put back on its original identity in under a second through the new operations, with nothing deleted.

What I'd tell another team

Those odd-looking records were how the system remembered requirements nobody had written down.

• Audit before you clean. Check every store that reads the data, and every lookup that resolves it.
• Separate logical identity from physical identity. If history joins on a hardware ID, the first refurbished unit that changes customers leaks data between accounts.
• Readers first, uniqueness last. With vendored shared code and a lagging replica, the order of the migration matters as much as the schema.
• Log changes where they happen. A trigger-written audit log covers every writer, including the ones nobody remembers, and it's there when you need to reconstruct what happened.
• Prevent mistakes where the context is. The installer standing at the pole knows whether it's a replacement. A heuristic running three days later doesn't.

When this is the wrong model: if history belongs to the unit, as in a warranty or repair workflow, the physical unit is the identity and the model inverts. If swaps are rare, a careful one-time cleanup may be all you need. And if nobody can freeze the legacy writers during the relabel, the completeness check undoes itself.

Related

• Distributed systems services: durable data models and migrations your team can run after I leave.