Which LLM understands Persian product search best? A real benchmark

Public LLM benchmarks are almost always English, and almost always synthetic. We needed something else: how well does a model understand what a Persian-speaking buyer is actually asking for? Here are the results of testing five models on real data.

387

real Persian searches from our own logs

112

that had shown the user a blank page

5

models, all on one identical prompt

$0.26

total cost of the benchmark

The task: which listing is actually the product?

When someone searches for «واسکازین ۹۰W140» (a gear oil grade), our crawler returns 20-24 listings from different Iranian shops. Some are gear oils, some are engine oils, some are unrelated. The model's job is to keep only the genuine product and drop the rest — replying with a strict JSON contract and nothing else.

There are two ways to fail, and both hurt the user: be so strict you keep nothing (a blank page, even though the product was right there in the list), or so loose you keep everything (which is not filtering at all).

The data: 112 searches that really did come back empty

Rather than inventing test cases, we pulled 387 distinct Persian searches from our own logs — each with the exact candidate list the model saw at the time. We focused on the 112 where the filter had kept nothing in production, i.e. precisely where a user saw a blank page. We also ran 60 healthy searches separately, to check that a model does not simply become permissive.

The queries are real, and full of the noise natural language actually has: misspellings («واسکارین» for «واسکازین»), a wrong letter in a part code (Isuzu "MPR" where the truck is the NPR), and a wrong unit («لامپ مهتابی ۱۵ آمپر» — lamps are rated in watts, not amps).

Results

That score is not a natural measure — it's a composite we built so the four columns below can be read at a glance. Look at what it does to the two middle models: DeepSeek rescues nearly twice as many blank searches as Gemini, but it is slower and returns 3% malformed replies, so it lands lower overall. Change the weights and the order changes; the raw numbers are right below.

The ranking

• 1) gpt-5.6-luna — 79. The clear winner: three times Gemini's rescue rate while staying tighter on healthy searches, with no malformed replies at all.
• 2) gemma-4-31b (free) — 71. The best free option by a wide margin: 65% rescued, though looser — it kept the entire candidate list 15% of the time.
• 3) gemini-3.5-flash-lite — 61. By far the fastest (1.4s) and clean, but far too strict: it still left three quarters of those searches blank. Its score comes from speed and well-formed output, not from rescues.
• 4) deepseek-v4-flash-0731 — 53. The cheapest, and ahead of Gemini on rescues; but not dependable: 3% of replies were unreadable (sometimes empty), it was the slowest usable model, and it over-kept more than anything else.
• 5) nemotron-3-ultra (free) — zero. Effectively unusable here: 105 seconds for a single call, and it answered in prose instead of the requested JSON.

Which model is "best" depends on which failure you fear more. The top two lead either way; after that it is a judgement call — if a blank result is the worse outcome, put DeepSeek above Gemini; if irrelevant clutter is worse, Gemini's precision at 1.4 seconds is still a defensible choice.

Two things we expected to help, that didn't

First, we added a prompt clause telling the model the user may have typo'd a brand or written the wrong unit, and to resolve what they evidently meant. It was written specifically for the three cases above. Across 112 searches it improved 17 and degraded 15 — noise — and on those three target cases it changed nothing at all. We removed it.

Second, we pinned temperature to 0 to make the classification reproducible. It is the right setting for this kind of task, but it did not deliver reproducibility: running each search three times, 33% of results differed between runs at temperature 0.7, and 43% at temperature 0.

The shared lesson: for this task, choosing the model mattered far more than tuning the prompt or the temperature. The gap between the best and worst usable model was 3×; the gap between the best and worst prompt was indistinguishable from noise.

Limits of this benchmark

• It measures one specific task — Persian product matching with a strict JSON output — not general Persian language ability.
• Every model got the same prompt at temperature 0; each would likely do better with a prompt tuned to it.
• Apart from the reproducibility test, each search ran once.
• Calls went through OpenRouter, which can route one model to different providers; some of the instability we measured may come from that rather than the models.
• The queries come from procurement: car parts, oils, filters, promotional and office supplies.

Total cost: $0.26

The whole benchmark — roughly 700 calls across five models — cost 26 cents. If you are choosing a model for your own Persian-language workload, measuring it on your own data is nearly free, and the answer can be a 3× difference in quality.

This filter is what sits behind our live price pages: it takes the raw shop listings and shows only the product you actually asked for.