Half the models failed our benchmark. The bug was ours.

Three months ago we built an AI filter whose job is to look at the raw listings our crawler brings back and keep only the product the user actually asked for. This week we benchmarked twelve language models on that task to decide which one to run. Eight produced broken output. Our first guess was that those eight were simply weak models. That guess was wrong.

12

models, one real search, one identical prompt

8

that returned at least one unusable reply

15%

failure rate this caused in our own fallback chain

0%

the same rate after a one-line change

The symptom: an empty reply, and no error at all

The failing models did not return an error. The connection was fine, tokens were consumed, cost was billed — and the content of the reply came back completely empty. To our code that reads as "the filter did not answer", and in that case we fall back to showing the raw list of listings. So the failure did not surface as an error message; it surfaced as a cluttered set of results.

When we pulled the per-call detail out of the provider's logs, three numbers told the whole story: output tokens exactly equal to the cap we set, reasoning tokens at almost the same figure, and a finish reason of "length" rather than "stop".

The model had spent the entire budget we gave it on thinking, and by the time it came to write the answer there was nothing left. Newer models reason to themselves before replying, and that reasoning is billed against the same output allowance as the answer. Our cap was 512 tokens — plenty for a short JSON reply, and not nearly enough for a short JSON reply plus several hundred tokens of deliberation.

The test: the same calls, with four times the ceiling

We replayed exactly the same requests with a 2048-token cap. Nothing else changed: same prompt, same candidate list, same temperature of zero.

Four of the eight "weak" models were fixed by one number in a config file. A ceiling of 5000 rescued two more, and that is what we shipped — because we measured it and a higher ceiling costs **nothing** for a model that does not need it: over all 152 searches, luna is identical at 2048 and 5000 (251 vs 250 reasoning tokens, $0.000607 vs $0.000606). A budget is a ceiling, not a spend. The remaining three are not fixable at any budget.

The uncomfortable part: this was live in our own system

Our primary model does not reason, so the bug never fired on the main path. But the fallback chain — the one that takes over when the primary is unavailable — was headed by a model that does reason. In other words, the bug was reserved for exactly the moments when things were already going wrong.

To measure it, we replayed 152 real recorded searches from our own logs through that model:

• At a 512-token cap: roughly one reply in seven came back unusable (15%), and the user saw an unfiltered list.
• At 2048: zero.

This also means our previous report was partly wrong. In it we said the best model rescued 78% of the searches that had come back empty. That figure was measured at the same 512-token cap. With the ceiling corrected, the real number is 96%. Eighteen percentage points we had attributed to the model were our own configuration.

Rescue rate, after the fix

The result that mattered most to us: the most expensive option on the list was also the weakest. gemini-3.5-flash-lite costs three times what gemma does and rescues a third as much. Its one real advantage is speed — and speed can be bought elsewhere in the chain, not at the point where result quality is decided.

The second lesson: valid output, wrong number

One model scored full marks on every output-format test, and then wrote in its Persian summary that the cheapest price was 74,500,000 toman — naming the shop. That shop's actual price, in the very list the model was given, was 75,450,000. The digits had been transposed. It also quoted another seller at 77,990,000 where the real figure was 77,999,000.

For a price comparison site this is the worst class of error: the answer looks entirely credible, the format is correct, the shop is real, and only the number is wrong. We dropped that model. We now check every figure a model writes in prose against the actual prices from that same search, automatically.

If you are building something similar

• The output token cap includes reasoning tokens. If the model reasons, the budget you meant for the answer never reaches the answer.
• Treat an empty reply as a distinct failure, not as "the model said nothing". We logged two very different situations identically, which is why this went unseen for months.
• Price per million tokens does not predict what a request costs. A model billing a tenth of the going rate can cost several times more per request, because it thinks for hundreds of tokens before every answer.
• Do not mistake a valid format for a correct answer. JSON validation tells you the structure is sound; it says nothing about the numbers inside it.

The method and dataset here continue our earlier benchmark on 112 Persian searches. This is the same filter that sits behind the results on live prices.