Why Detection From the Air Is a Different Problem
Object detection models trained on ground-level imagery collapse when you point the camera down from 200 feet. The failure is not the model architecture. It is that almost every assumption baked into ground-level datasets breaks.
A person standing 15 feet from a car-mounted camera occupies several hundred pixels vertically. The same person at 250 feet altitude with a 1-inch sensor and a moderate lens occupies perhaps 20 pixels, and small-object detection is a known weak point in convolutional and transformer detectors alike. Add the rest of the aerial penalty stack: scale changing by a factor of five as altitude varies, viewing angle shifting from oblique to straight down, vibration and rolling-shutter skew smearing edges, sun and deep shadow within one frame, and cluttered terrain instead of a tidy indoor scene.
Two consequences follow. First, ground-view pretrained weights are a starting point, not a solution. Second, the sensor and optics decide more of your detection range than the model does. Ground sample distance is the governing number: a target needs roughly 20 to 30 pixels across its smallest dimension for reliable classification, which sets your maximum altitude for a given focal length and pixel pitch long before any software choice matters. The sensor tradeoffs are covered in choosing an image sensor, and for many missions the more useful answer is a second spectrum entirely, as discussed in thermal camera drone payloads.
Three Tasks People Blur Together
Detection answers what is in this frame and where, as a bounding box per object, independently on every frame. It is stateless. A detector running at 15 frames per second gives you 15 unrelated opinions per second, with objects appearing and vanishing frame to frame.
Tracking adds identity across time. The same vehicle keeps its ID as it moves, is briefly occluded by a tree, and reappears. Tracking turns detections into counts, trajectories, speeds, and dwell times, and it suppresses flicker: a box that appears once is noise, one confirmed across ten consecutive frames is real. Most systems pair detection with a lightweight association step driven by motion prediction, and the hard part is re-identification after occlusion when a dozen similar targets are in frame.
Segmentation labels every pixel instead of drawing boxes, which is what area measurement needs: crop stress coverage, roof damage extent, flooded acreage, stockpile boundaries. It costs considerably more compute, so choose it deliberately rather than defaulting to the heaviest option.
From Pixel to Map Coordinate
A bounding box is worthless to an operator. What they need is a location. Converting one to the other requires more than the detection.
You need calibrated camera intrinsics including lens distortion, gimbal orientation at the moment of exposure, aircraft attitude and position from the flight controller, and a terrain model to intersect the camera ray against. Assuming flat ground is the usual shortcut, and on a hillside it can put a target tens of meters from its true position.
The failure that bites hardest is time synchronization. If image timestamps and position or attitude timestamps come from different clocks, a drone moving at 10 meters per second accumulates a meter of georeferencing error per 100 milliseconds of skew. Hardware-triggered exposure timestamps tied to the flight controller clock solve this; software timestamps taken when a frame reaches userspace do not. The same discipline underpins drone photogrammetry and aerial mapping.
Where the Inference Runs
Three options, and the choice is driven by latency, link reliability, and watts.
Onboard. A compute module on the aircraft running the model. Latency is a few tens of milliseconds and the system keeps working when the radio link drops, which is the only acceptable answer for anything closing a control loop, such as visual tracking of a moving subject or the perception feeding obstacle avoidance. The cost is power and mass: an embedded GPU module drawing 10 to 25 watts under load is a real fraction of a multirotor's budget and measurably shortens the endurance figures in drone battery systems. Dedicated accelerators draw a few watts for far less flexibility. Thermal design matters more than teams expect, since a module that throttles in still air behaves nothing like it did on the bench with a fan.
Ground station. Stream video down and process on a laptop or edge box. Unlimited compute and no flight penalty, but you inherit the link budget, compression artifacts that hurt small-object detection, and 150 to 400 milliseconds of latency. Fine for operator review, unacceptable for autonomy.
Cloud, after the flight. Correct for mapping and inspection reporting, where the answer is a deliverable rather than a reaction. Run the heavy model on full-resolution imagery once the aircraft lands.
A hybrid usually wins: a small onboard detector flags candidates and drives the gimbal, while full imagery is kept for a thorough offline pass.
The Dataset Is the Actual Asset
Model architectures are commodities. A team's advantage is its labeled imagery from the altitudes, angles, seasons, and lighting conditions its customers actually fly in. Budget for it as a line item rather than treating it as a task someone does between sprints.
What a usable aerial dataset needs: imagery across the full altitude band you will operate in, not just the convenient one; nadir and oblique angles; overcast, direct sun, and low-angle light; the terrain and vegetation of your deployment region; and hard negatives, the things that look like your target and are not. Most field false positives come from a confuser class nobody thought to label. Synthetic data helps for rare targets but carries a domain gap only real imagery closes. Plan on continuous collection with field failures routed back into training, the operating model in adding AI to a hardware product.
Measuring the Thing That Matters
Mean average precision on a held-out split is a research metric and it will mislead you. Operators care about different numbers: detection probability for a specified target size at a specified altitude, false alarms per hour of flight, time from target entering the frame to alert reaching the operator, and track continuity through occlusion.
False alarm rate decides whether the system gets used at all. An operator receiving 40 spurious alerts per hour stops reading them, and the detector becomes worse than nothing, so teams tune the confidence threshold and the multi-frame confirmation count against a false alarm budget rather than a precision-recall curve. Where a miss is unacceptable, as in search and rescue drones, the balance moves the other way and human review absorbs the extra alerts.
Get the Vision Stack Specified Correctly
Sensor choice, altitude band, compute placement, and dataset plan are one coupled decision, and getting them out of order is how vision programs burn a year. Projects House works with drone teams on the whole chain, from optics and payload integration to onboard inference and georeferencing accuracy. Tell us what you need to detect and from how far through our contact form.