A lab set the most advanced AI on the market to pilot an office drone through five chained stages. It beat the human benchmark in every single one of the five. The complete sequence, start to finish, works 2.8% of the time.
Both numbers are true at the same time, and the gap between them is the most useful thing to come out of an AI test this month. It's what explains why so many automation projects impress in the demo and disappoint on Monday.
What was tested
Andon Labs, a lab that evaluates models on long tasks, built Drone-Bench: the model writes the code that makes a cheap drone, a DJI Tello EDU, carry out a mission on its own. There are five stages, each feeding the next:
- rebuilding the office as a three-dimensional model from video;
- figuring out where the drone is within that model;
- navigating without crashing;
- detecting the right person;
- following that person.
GPT-6 Astra, released on September 3, was the first model to beat the human-AI baseline in every one of the five stages. That's the result that made headlines, and it's real.
The rest of the report is less celebratory. Beating the baseline happened on at least one attempt per stage. Counting all ten rounds, person detection succeeded in 4; the 3D reconstruction, in 1. The lab sums it up in one line: best-case scoring doesn't mean reliable performance.
Why five good stages don't make a good process
Reliability doesn't add up, it multiplies. For the whole mission to succeed, all five stages need to succeed in the same round. Just multiply the two published rates to see what happens: 4 in 10 times 1 in 10 already comes out to 4%. With the other three stages along the way, the measurement for the entire sequence came in at 2.8%.
The same math applies to any automated process, drone or no drone. If each stage succeeds at a given rate, the process succeeds at the product of those rates:
- 5 stages at 95% success each: 77% of full processes correct;
- 5 stages at 90%: 59%;
- 5 stages at 80%: 33%;
- 10 stages at 90%: 35%.
And the reverse path is even more alarming: for a five-stage process to deliver 95% end-to-end accuracy, each stage needs to succeed 99% of the time. That's not a target you hit by tweaking a prompt's wording.
How many stages does your process have
The math only works if you count the stages for real. A stage is any point where the output of one thing becomes the input of the next. A customer service interaction that looks like a single task is usually six:
- reading the customer's message;
- understanding what the request is;
- checking inventory or the schedule;
- calculating price, timeline, or shipping;
- writing the reply;
- logging it in the system.
A billing request, a quote, and a resume screening have similar structures. If each of those six stages succeeds 90% of the time — which already sounds good — the entire interaction comes out correct in 53% of cases. This isn't hypothetical: in OpenAI's own survey, more than half of long-running tasks needed human intervention. Half of customer interactions have something wrong somewhere along the way, and most of those errors are silent: the price came from the old table, the deadline ignored a holiday, the record never got logged.
The move that raises the outcome the most
Since the per-stage target is too high to reach by tweaking prompts, the answer is to change the makeup of the chain itself. In order of impact:
1. Replace a model-driven stage with a code-driven stage whenever the rule is fixed. Calculating shipping, applying a price table, checking whether a date is a business day, adding tax: these are rules, and a rule written in code succeeds 100% of the time. Every stage that moves from the model to code drops out of the multiplication. That's the difference between 90% and 100% in one of the factors, and the effect on the final outcome is large.
2. Cut the number of stages. Two stages that always happen together can become one. A stage that exists only because the old system required it usually can be removed with no downside.
3. Put human review before the irreversible step. Sending to the customer, charging, publishing, deleting. Review doesn't improve the multiplication math: it stops the error from leaving the company and lets you restart from the middle, not from the beginning.
4. Favor stages that fail loudly. A stage that errors and flags it costs far less than one that errors silently. When the model isn't sure, it needs a way to say so instead of inventing a plausible value.
What this test doesn't tell you
Piloting a drone is harder than handling a customer: it involves the physical world, a camera, network lag, and zero chances to try again after hitting a wall. An office's success rates aren't your customer service's success rates, and it would be dishonest to transplant that 2.8% onto your company.
What does transfer is the structure of the math. And it's worth noting that Andon Labs projects a frontier model will solve the entire sequence in a single attempt by the first quarter of 2027. That's a projection, not a result, based on the pace of the last two years — but it points to the direction: per-stage rates keep rising, and the multiplication keeps getting less brutal.
Until that happens, the practical decision stays the same. Before automating a process, count the stages and measure each one separately. It's the only way to know whether a bad outcome came from one weak stage or from many mediocre ones — and the fix is completely different in each case.
How we handle this in practice
When we implement AI in business processes, the first step isn't picking a model: it's mapping out the chain, separating what's a rule from what's a judgment call, and measuring stage by stage. The gain almost always comes from removing stages from the AI, not adding more.
Sources
Drone-Bench and Vending-Bench results as published by Andon Labs and covered by The Decoder, September 2026. Per-subtask rates and the full-sequence rate are from the lab's report. The GPT-6 Astra launch on September 3, 2026 is from OpenAI.
Frequently asked questions
What does it mean that the AI only nails the drone test 2.8% of the time?
It means the complete mission, with all five chained steps succeeding in the same run, happened in about 2.8% of the attempts. That coexists with the fact that the model matched or beat the human benchmark on every individual step at least once — beating the benchmark happened on at least one attempt per step, just not on all of them simultaneously.
Why does a process with reliable steps still fail as a whole?
Because step reliability multiplies instead of adding up. Five steps that each succeed 90% of the time deliver a fully correct process only 59% of the time. With ten steps at 90%, that drops to 35%. The longer the chain, the bigger the gap between how good each part is and how good the whole thing turns out.
How many steps does a typical customer service process have?
Usually six: reading the message, understanding the request, checking stock or the schedule, calculating price and delivery time, writing the reply, and logging it in the system. A step is any point where one thing's output becomes the next thing's input, and the count usually surprises people who assumed it was a single task.
How can you raise the success rate of an automated process?
The highest-impact move is swapping a model-driven step for a code-driven one whenever the rule is fixed, like calculating shipping or applying a price table, because code gets it right every time and drops out of the multiplication. After that, focus on cutting the number of steps, adding a human check right before the irreversible action, and favoring steps that flag their own mistakes.
Does human review improve the success rate?
It doesn't change the multiplication math, but it does change the damage. It keeps the error from reaching the customer and lets you restart from the middle of the process instead of the beginning. That's why it belongs right before the irreversible step: sending, charging, publishing, or deleting.
Do these numbers apply to my business?
The rates don't, but the underlying math does. Flying a drone involves the physical world, a camera feed, and network lag, making it harder than replying to a customer. What actually carries over is the method: count the steps in your process, measure each one separately, and multiply them together to find out whether the problem is one weak step or several mediocre ones.