The System That Worked Perfectly Until It Succeeded

The backend that carried fifty pilot units through a beta program was probably one virtual machine running an API server and a Postgres instance. It was fast, it cost $40 a month, and it never gave anyone trouble. Then distribution picked up, the fleet crossed a few thousand units, and the dashboard started taking eleven seconds to load. Alerts arrive twenty minutes late. The nightly report job now overlaps the next night's run. Nothing is technically down, so nobody can point at a single failure, and every week is slightly worse than the last.

This is the ordinary trajectory. IoT load does not grow like web traffic. Web traffic follows human attention and drops at night. Device traffic is relentless, roughly linear in fleet size, and heavily write-biased. A fleet of 5,000 units each reporting every 30 seconds generates about 14.4 million writes a day, at essentially constant rate around the clock. Nothing about that resembles a content site.

Where Systems Actually Break

  • The database, first and hardest. A single table of sensor readings with an index on device and timestamp behaves beautifully until it passes a few hundred million rows, and then index maintenance, vacuum, and disk throughput all turn against you at once. The failure mode and the fix are described in time-series databases for sensor data.
  • Connection handling. Ten thousand devices holding persistent connections is a very different problem from ten thousand HTTP requests spread over an hour. File descriptor limits, TLS session memory, and keepalive traffic become the constraint before CPU does.
  • Synchronous ingest. If a device write goes straight into the database and the device waits for the commit, database latency becomes device latency, and a slow query turns into a fleet-wide retry storm.
  • Thundering herds. Every device rebooting after a regional power event, or every device firing its report on the exact minute boundary, produces a spike ten to fifty times the average rate. Average capacity planning does not survive it.
  • The dashboard. Aggregate queries scanning raw readings on every page load will outgrow the ingest path as a source of database pressure. What the console has to show, and therefore what has to be precomputed, is covered in the IoT device dashboard.
  • Operations, not code. At a few hundred devices you fix problems by hand. At ten thousand you cannot, and the practices in IoT fleet management become the actual bottleneck.

Early Decisions That Buy Scale Cheaply

Put a queue between ingest and storage. A broker or stream, Kafka, Kinesis, or a managed MQTT ingest tier, lets the device write land in single-digit milliseconds while a consumer batches inserts. This one change absorbs spikes, decouples device availability from database health, and typically costs a week to implement at the start and two months to retrofit.

Choose the protocol on cost, not familiarity. An MQTT publish carries roughly 2 to 4 bytes of protocol overhead against several hundred for an HTTPS request with headers and a fresh handshake. At fleet scale that difference shows up on your cellular bill and on your ingest tier, which is why the comparison in MQTT vs HTTP is a budget decision as much as a technical one.

Design the data retention policy before the first byte. Raw at full resolution for 30 days, one-minute rollups for a year, hourly rollups forever, cold storage in object storage after that. Deciding this later means backfilling billions of rows.

Make services stateless. Session state in process memory is what prevents you from adding a second instance behind a load balancer.

Give every device a unique credential from day one. A shared key across the fleet cannot be rotated without a recall, and per-device certificates issued on the line are the pattern in secure device provisioning.

Ship OTA before you ship hardware. Every scaling problem eventually needs a device-side change: a longer reporting interval, backoff with jitter, a smaller payload. Without OTA firmware updates your only lever is the cloud, and that is a very expensive way to solve a firmware problem.

The Warning Signs

Scaling problems announce themselves months ahead if you are measuring the right things. Track p95 and p99 latency rather than averages, because the mean stays flat while the tail degrades. Watch ingest lag, the gap between device timestamp and stored timestamp, which is the earliest honest indicator of trouble. Watch database CPU during your daily peak, not the daily average; if peak is above 60 percent you have a quarter or two of headroom, not a year. Watch the ratio of infrastructure cost to active devices; if cost per device is rising rather than falling as the fleet grows, something in the architecture is superlinear and it will not fix itself. Track the deploy-to-recover time on a bad release, because that number, not raw throughput, is what customers experience during an incident.

Grow in Stages, Not in One Leap

Resist the urge to build for a million devices while you have four hundred. Over-architecting is its own failure: a microservice mesh with five datastores maintained by two engineers will fail more often than a monolith, and the monthly bill described in IoT cloud infrastructure costs arrives whether or not the fleet is there yet.

A workable staging is roughly this. Up to a few hundred devices, one managed database and one application tier, with the queue already in place. Into the low thousands, split ingest from the API, add a time-series store and precomputed rollups, and move the dashboard onto the rollups. Into the tens of thousands, partition by device or region, add read replicas, cache aggressively, and automate deploys and rollbacks. Above that, the questions become regional isolation, multi-tenancy boundaries, and whether your managed platform's device registry still fits, which is where the platform comparison in AWS IoT vs Azure IoT starts to matter concretely.

The practical rule: build the next stage when you are at roughly half its trigger point. That is early enough to migrate calmly and late enough that you are not paying for capacity you do not use.

Design the Backend Your Fleet Will Need

Projects House architects connected-product backends alongside the firmware and app that feed them, sizing for the fleet you expect rather than the one you have. Tell us your device count, reporting interval, and growth plan through our contact form and we will map where your current stack runs out.