Published August 13, 2026

15 language models, two real jobs: which ones are worth it?

Every price search in our product makes two language-model calls. We benchmarked 15 models on 152 real recorded searches for both, measuring success rate, overall score and real cost. The most expensive option on the list came last on quality.

Every price search on Gheymatshenas needs a language model twice, and the two jobs are nothing alike. First the answer loop: decide what to go and look for in the shops, then write the reply in Persian. Second the relevance filter: take the 20–24 raw listings the crawler returns and keep only the genuine product. We benchmarked 15 models on our own real data for both.

15
models, one identical prompt: 152
real recorded searches: 2
distinct jobs per search: 30x
cost spread, first to last

Job one: the relevance filter

This call runs on every search and carries the volume. The headline measure is simple: of the searches that had actually shown a user "nothing found", how many does the model rescue? The second measure matters because the opposite failure is just as bad: how often did it keep the entire list, i.e. not filter at all?

Bar chart of score and cost for all 15 models on the relevance filter task — Score out of 100 over 152 real searches. The right-hand column is the cost per million calls.

Model	Score	Success rate	Kept whole list	Latency	Cost / 1M calls
gpt-5.6-luna	97	96%	5%	568ms	$607
ling-3.0-flash	95	100%	10%	908ms	$315
gpt-5.4-nano	93	80%	0%	594ms	$548
gemma-4-31b-it	86	71%	15%	604ms	$334
gemma-4-31b-it:free	81	69%	17%	1371ms	$0
gemini-3.1-flash-lite	80	60%	0%	1804ms	$0
deepseek-v4-flash-0731	78	90%	0%	1288ms	$1,015
gemini-3.5-flash-lite	71	30%	0%	1345ms	$0
gemini-3.5-flash-lite (OpenRouter)	71	26%	8%	736ms	$879
deepseek-v4-flash	70	48%	21%	1399ms	$514
qwen3.7-flash	23	—	—	978ms	$597
hy3-preview	0	—	—	2752ms	$1,260
mimo-v2.5	0	—	—	1344ms	$2,099
nemotron-3.5-lightning	0	—	—	514ms	$0
gemma-4-31b-it (Google direct)	0	—	—	36,308ms	$0

Score = 30% clean output + 35% success rate + 20% strictness + 15% speed. Cost is deliberately kept out of the score so the trade-off stays visible. "Kept whole list" — lower is better. The last five never produced valid JSON at all, so a success rate is meaningless for them.

Two things in that table surprised us. The cheapest paid option (ling) has the highest success rate — the only model that rescued all 109 blank searches it saw. And the most expensive option that works at all (gemini-flash-lite bought through OpenRouter, $879) has the lowest: 26%. Nearly three times the money for a quarter of the result.

The last five could not produce valid JSON at all. Three of them keep "thinking" up to whatever ceiling you give them and never reach the answer — and two of those are among the most expensive rows in the table. They charge for failing.

Job two: the answer loop

Here the model has to call the search tool correctly and then write the Persian answer. We measure two things no public benchmark reports: whether every number it writes really appeared in the price list it was shown, and whether it fires one search per question rather than several.

Bar chart of score and cost for models on the answer loop task — Score = 45% number fidelity + 30% correct tool decision + 15% single call + 10% speed.

Model	Score	Number fidelity	Tool decision	Calls per question	Latency	Cost / 1M searches
gemini-3.5-flash-lite	100	2 of 2	3 of 3	1	1.4s	$0
gemma-4-31b-it	90	5 of 5	2 of 3	1	1.8s	$884
gpt-5.6-luna	85	5 of 6	3 of 3	1	9s	$1,028
deepseek-v4-flash	80	9 of 9	3 of 3	3	21s	$1,866
ling-3.0-flash	70	9 of 9	2 of 3	5	9s	$294
gpt-5.4-nano	65	4 of 9	3 of 3	1	14s	$2,273
gemini-3.1-flash-lite	—	not measured	3 of 3	1	1.7s	$0

"Number fidelity" is how many of the figures the model wrote in its Persian prose really appeared in that search's price list. "Tool decision" was probed on three questions: a product, a car and an Arabic query. This table rests on much weaker evidence than the one above — three tool probes and one prose sample per model.

Two models were ruled out here for reasons no public benchmark reports. ling fired five searches for one question and invented a Peugeot 207 for a query about a Pride 1400 — and each search is a live crawl on the user's own machine. gpt-5.4-nano scored full marks on output format and then wrote that the cheapest price was 74,500,000 toman when that shop's real price, in the very list it was given, was 75,450,000. For a price comparison site that is the worst kind of error: everything looks right and only the number is wrong.

Price per token does not predict what a request costs

Column chart of cost per million calls for five models — What a million filter calls actually cost, measured on real token counts.

This was the most useful practical lesson. A model billing a tenth of the going rate per token can cost several times more per request, because it spends hundreds or thousands of tokens "thinking" before every answer and those are billed too. The only way to know is to send your own real prompt and read the invoice.

What we picked

For the relevance filter: ling-3.0-flash — the only model with a 100% success rate, at half the price of the next one.
For the answer loop: gemini flash-lite — one call, fastest, and its numbers were exact. Models that fire several searches per question have no place here.
Ruled out: gpt-5.4-nano for the wrong price, and the five models that never produced valid output.

Limits of this benchmark

It measures two specific jobs, not general model ability.
Every model got the same prompt at temperature zero; each would probably do better with a prompt tuned to it.
The filter table rests on 152 searches; the answer-loop table on three tool probes and one prose sample — far weaker.
Each search was run once, and we have previously measured that a model does not always give the same verdict on identical input.

This continues our earlier benchmark on 112 Persian searches, and the token-budget bug report explains why half of these models looked broken at first. You can see the result on live prices.