Why Products Freeze in the First Place

A device that has run flawlessly on the bench for six weeks will still lock up in the field, and the causes are almost never the ones caught in code review. A cosmic-ray or ESD-induced bit flip corrupts a program counter. A heap fragments after four months of uptime and an allocation fails in a path nobody handled. Two tasks deadlock on mutexes taken in different orders, a race that needs a specific interrupt timing that occurs once every few hundred thousand hours across a fleet. An I2C sensor holds SDA low forever after a brownout and your driver spins in a blocking wait.

None of these are fixable by writing better code, in the sense that no realistic amount of care drives the rate to zero. Across a fleet of ten thousand units running continuously, a hang rate of one per unit-century still means a device freezes every few days somewhere. If the product is a mounted sensor in a customer's ceiling, that is a truck roll. Recovery is not a fallback for sloppy firmware; it is a required feature of any product that ships.

How a Watchdog Actually Works

A watchdog is a counter running off its own clock that resets the processor when it reaches its limit. Firmware must periodically clear it, which is called kicking, feeding, or servicing the watchdog. If the code stops running, stops looping, or gets stuck, nothing clears the counter and the chip resets.

Two kinds matter. The internal watchdog is a peripheral inside the microcontroller, free, and configured in a few lines. Its weakness is that it shares the die: a clock failure or a latched-up core can take the watchdog with it. The external watchdog is a separate supervisor IC costing $0.30 to $1.50 that holds the reset line and cannot be disabled by confused firmware. For anything unattended, safety-relevant, or expensive to visit, fit the external part. It is the cheapest reliability insurance on the bill of materials.

Configuration details decide whether it helps or hurts:

  • Timeout selection. Long enough that the slowest legitimate loop iteration, including a flash write or a blocking radio call, comfortably fits. Short enough that a customer does not notice. One to eight seconds covers most products; a motion-control loop may need tens of milliseconds.
  • Windowed mode. A window watchdog rejects a kick that arrives too early as well as too late, which catches runaway code that happens to loop through the kick instruction. Use it where the part supports it.
  • Never kick from a timer interrupt. This is the single most common mistake in production firmware. An interrupt keeps firing while the main application is deadlocked, so the watchdog is faithfully serviced by a dead product.
  • Kick from evidence, not from habit. In an RTOS, have each critical task set a flag on each successful cycle, and let one supervisor check that every flag has been set within its expected period before it kicks. Now the watchdog monitors the whole system rather than one loop. Which tasks exist and how they are scheduled follows from the choices in RTOS vs bare metal, and both major kernels compared in FreeRTOS vs Zephyr ship task-watchdog facilities worth using instead of rolling your own.
  • Enable it at boot, in hardware if possible. Many parts offer a fuse or option byte that starts the watchdog before the first instruction, closing the window where a corrupt image never enables it.

Recovery Is More Than a Reset

A reset that returns the device to the same state that hung it produces a boot loop, which from the customer's side is worse than a freeze because the LED flickers and the product looks possessed. Real recovery has three additional parts.

Know why you reset. Read the reset cause register at boot: power-on, brownout, external pin, software, or watchdog. Store it in a small non-volatile record along with a boot counter, the firmware version, and, if you can afford the flash, a crash snapshot with the program counter, the fault registers, and the last task that ran. Without this, field failures are unexplainable and you will chase them for months.

Escalate. Behave differently on the fifth watchdog reset than on the first. A workable ladder: normal boot; then boot with radios and optional subsystems disabled; then clear the configuration region and boot with defaults; then fall back to the last known good firmware image. That last rung requires a dual-slot layout, which is a design decision made in the bootloader and is nearly impossible to add after production.

Back off before retrying. A device that resets every three seconds and immediately hammers the network is worse than one that waits. Add exponential backoff with jitter, both to protect battery and to keep a fleet-wide event from becoming a denial-of-service against your own servers.

Report what happened. On the next successful connection, upload the reset cause and counters. Reset telemetry across the fleet is the earliest signal that a release has a problem, and it is what tells you whether to halt a rollout. Fixing the root cause afterward requires a working OTA update path, because a recovery mechanism only buys you the time to ship a fix.

The Other Layers of Defense

The watchdog is the last line, not the only one. Add a brownout detector configured above the minimum operating voltage so the part resets cleanly instead of running on a sagging rail. Handle hard faults and memory management faults with a handler that records context and resets deliberately rather than spinning in a default infinite loop. Give every blocking peripheral operation a timeout, and implement the I2C bus recovery sequence of clocking SCL until a stuck slave releases SDA. Enable the memory protection unit if the part has one, so a wild pointer traps instead of corrupting a neighbor's stack. Fill unused stack with a pattern and check the high-water mark, because silent stack overflow causes many of the faults people blame on hardware. Validate configuration data in flash with a CRC. None of this is exotic, and together it moves most field hangs into categories the watchdog never has to handle.

Proving It Actually Works

An untested watchdog is a comment. Test it deliberately. Add a debug command that enters an infinite loop and confirm the reset time matches the configured timeout. Add one that disables interrupts and spins. Verify the reset cause register reports correctly for each path. Confirm the recovery ladder escalates by forcing consecutive resets, and confirm the counter clears after a period of healthy operation, otherwise a device that hangs once a year eventually reaches the bottom rung for no reason.

Then test the environment rather than the code. Run power-cycle soak with a programmable supply that interrupts at random points, including partial brownouts, for thousands of cycles. Run a multi-week soak on several units at temperature extremes, watching for drift, memory growth, and counter rollovers, which is what the protocols in reliability testing are designed to catch. Verify behavior with the debugger detached, because many parts suspend the watchdog while one is connected, and teams have shipped products whose watchdog was never once observed to fire, a trap familiar to anyone who has worked through embedded debugging tools. Finally, confirm the recovery path does not wreck battery life; a device that resets and rejoins the network aggressively can burn a year of budget in a week, which is why the sleep discipline in low-power firmware has to cover the recovery states too.

Build a Product That Recovers on Its Own

Projects House designs firmware with the failure paths specified alongside the features: watchdog strategy, reset diagnostics, staged recovery, and the field telemetry that tells you what happened. Describe your device and where it gets deployed through our contact form and we will review your recovery architecture.