What Flight Controllers Actually Do: Sensors, Signal Processing, and Vision

An inspection quadcopter flying inside a warehouse crosses under a steel mezzanine and its GNSS (global navigation satellite system) fix drops out completely. A cheaper flight stack would treat that loss as an emergency and trigger a failsafe landing on the spot. This one keeps flying, holding position and heading steady, because a downward-facing camera has been feeding the flight controller a second opinion on position the entire time, and the moment GNSS disappears, that second opinion quietly becomes the primary one. No alarm sounds, no mode switch flashes on a ground station screen. The handoff happens inside a single update cycle, and the pilot on the ground may not even notice it occurred.

Nothing about that handoff involves the motors, the propellers, or anything most people picture when they hear "flight controller." It happens entirely inside the part of the system that decides what to believe. That decision, made continuously and largely invisibly, is the actual core function of a flight controller, and it matters more to whether an aircraft stays in the air than any single sensor or any single line of control code.

It is easy to think of a flight controller as the box that keeps a multirotor stable, translating stick inputs or waypoints into motor commands. That part is real, but it is downstream of a harder problem: before the flight controller can decide what the motors should do, it first has to decide where the aircraft actually is, which way it is pointed, and how fast it is moving. That second question, which way it is pointed, is what the rest of this post calls attitude: the aircraft's orientation in three dimensions, described as roll (tilt side to side), pitch (tilt nose up or down), and yaw (rotation around the vertical axis, which way the nose is heading). No single sensor answers any of this well on its own. An IMU (inertial measurement unit) drifts. A GNSS receiver updates slowly and can drop out entirely. A barometer gets fooled by its own propwash. The flight controller's estimator has to merge all of it, continuously, into one answer it is willing to act on.

This is worth separating clearly from control, because the two problems get solved by different parts of the system and fail in different ways. A control loop failure usually looks like an oscillation or an overshoot, something visibly wrong with how the aircraft is moving. An estimation failure is quieter and more dangerous: the control loop can be working perfectly, faithfully steering the aircraft toward a position that the estimator has gotten wrong. The aircraft flies a clean, stable path to the wrong place, which is a much harder failure to notice in the moment than a visible wobble.

Picture the estimator as a judge listening to several witnesses who each saw part of the same event and none of whom agree completely. The IMU testifies fast and often, dozens of times a second, but its story drifts further from the truth the longer it talks uninterrupted. The GNSS receiver testifies rarely and slowly by comparison, but when it does speak it anchors the story to an absolute position. The barometer and magnetometer each add a partial account, altitude and heading, each with its own known blind spots. The judge's job is not to pick one witness and ignore the rest. It is to weigh every account by how reliable it has been recently and produce a single verdict, updated continuously, that is more accurate than any witness alone.

That judge, in both of the major open flight stacks, is built around an Extended Kalman Filter, or EKF. PX4's implementation requires a minimum viable set of testimony before it will even open the case: IMU data, a source of heading from either a magnetometer or an external vision system, and a source of height from GNSS, barometer, a rangefinder, or vision. Everything else, GNSS position, optical flow, additional IMUs, gets fused in on top of that minimum once it is available.

The filter does not treat every witness equally just because it is present. Each sensor's contribution is weighted by a running estimate of its own noise and reliability, so a magnetometer that has started producing inconsistent readings gets down-weighted automatically rather than trusted at face value until someone notices a problem. That weighting is what allows the estimator to degrade gracefully instead of catastrophically: losing a sensor does not mean losing the estimate, it means losing one witness's testimony and leaning harder on the others until it either recovers or gets replaced.

Each sensor earns its place in that fusion for a different reason, and each comes with a specific failure mode the estimator has to manage. The IMU is fast and always available, updating at a minimum of 100 Hz, which makes it the backbone of short-term attitude and rate estimation, but its bias drifts over time with nothing to correct it. GNSS corrects that drift with an absolute position fix, but it updates far more slowly, can be degraded by multipath in urban canyons, and disappears entirely indoors or under structure, exactly what happened to the warehouse inspection quadcopter. The barometer gives a cheap, continuous altitude estimate, but PX4's own documentation notes it is subject to errors from aerodynamic disturbances caused by the aircraft's own wind-relative velocity and orientation, which is a technical way of saying propwash can fool it mid-maneuver. The magnetometer supplies heading, but is vulnerable to magnetic interference from nearby structures or the aircraft's own wiring, the same kind of real-world detail that no spec sheet fully captures until it shows up on a real airframe, close to the lesson this series drew when the DJI Agras T50's payload forced a structural step-change no simulation would have flagged.

Vision changes the calculus rather than just adding another witness. A downward-facing camera running optical flow, or a full visual-inertial odometry setup, can supply position and velocity estimates with no dependency on GNSS at all, which is exactly why the warehouse quadcopter kept flying. This is not a niche feature bolted onto a subset of aircraft. PX4's estimator explicitly supports vision pose data as an alternative heading and position source, on equal footing with a magnetometer or GNSS, because for indoor inspection or dense urban delivery, GNSS is not a fallback problem to solve later, it is a sensor the mission has to plan around not having.

None of this comes for free computationally. Fusing more sensor streams means running more of the estimator's internal math on every update cycle, and vision processing in particular is demanding enough that it usually runs on a companion computer rather than the flight controller itself, feeding its output back in as just another input to the same EKF. That division of labor, cheap and fast sensors handled directly by the flight controller, expensive and rich sensors preprocessed elsewhere, is itself a design decision with the same redundancy and cost tradeoffs this series has been tracking since the constraint chapter.

The specifics of that decision shift noticeably between small and large platforms, and not just because bigger aircraft can afford heavier compute. A small racing-class or line-of-sight multirotor typically runs a single EKF instance against a single IMU, updates its control loop fast relative to how quickly the airframe can physically move, and skips a companion computer entirely, because the consequence of a bad estimate on a short, observed flight is low enough that the added weight and cost are not worth carrying. A large BVLOS-capable platform runs multiple EKF instances by necessity, the redundancy floor from Post 6 leaves no other option, and often adds sensors a small multirotor never carries at all, an airspeed sensor on a fixed-wing or VTOL airframe being the clearest example, since stall margin cannot be inferred from GNSS ground speed alone. The larger aircraft's control loop also has to account for its own greater mass and inertia, which respond to a control input more slowly than a small quad does, so the same estimator producing the same kind of answer still needs different tuning underneath depending on how quickly the airframe it is describing can actually move.

The interesting design decision is not which sensors to include. Most modern flight controllers can accept all of them. It is deciding which sensors are load-bearing for a given mission and which are supplementary, because that decision is really a restatement of the redundancy floor this chapter has been building toward since Post 6. A mapping platform flown over open farmland can treat GNSS as its primary position source and vision as an optional refinement. A warehouse inspection platform has that relationship reversed: vision is load-bearing, and GNSS, when it exists at all indoors, is the supplement. Getting that assignment backwards does not show up in a bench test or even a clean simulation. It shows up the first time the aircraft flies a real mission and loses the sensor everyone assumed was primary.

This is also why sensor fusion cannot be treated as a solved problem once the estimator is configured and flying smoothly in testing. A magnetometer that reads clean on an open test field can behave differently over a mission site full of rebar and structural steel, in the same way a bench-tested vibration profile can shift once an airframe is loaded and flying its actual route. Both failures share the same shape: a sensor that was reliable in one environment quietly becoming unreliable in another, with the estimator doing its best to compensate right up until it can't.

Ship navigators solved a version of this problem long before any of this hardware existed. Dead reckoning, tracking speed and heading over time from a known starting point, gives a continuous position estimate that drifts further from the truth the longer the ship sails without a fix. A star sighting or a lighthouse bearing corrects that drift with an absolute reference, but only occasionally, and only when conditions allow it. A skilled navigator was never choosing between dead reckoning and a star sighting. They were constantly blending both, weighting each by how much time had passed and how reliable the latest fix was. An EKF fusing an IMU with a GNSS receiver is running the same navigation problem, just resolved thousands of times a second instead of once a watch.

A flight controller's hardest job is not stabilizing the aircraft. It is producing a trustworthy estimate of position, velocity, and orientation from sensors that are each individually unreliable in different ways.

• No single sensor is trustworthy on its own: IMUs drift, GNSS drops out, barometers get fooled by propwash, magnetometers pick up interference
• An Extended Kalman Filter fuses these sources continuously, weighting each by its recent reliability rather than picking one to trust
• Vision-based navigation is a core estimator input for GNSS-denied missions, not a backup feature, and needs to be planned for as load-bearing from the start
• Redundant sensors from Post 6 only add real protection when something, like multiple EKF instances, is actively comparing them against each other
• The design decision that matters most is deciding which sensor a mission actually depends on, not how many sensors the aircraft carries

Every one of these sensors can be fused perfectly and an aircraft can still fail. The next post looks at what actually goes wrong when it does, and why so many of those failures trace back to a decision made long before the crash itself.

Curious to exchange some ideas? Reach out via the contact form or connect on Linkedin!