Nine days of Poolside's free model in production

Gheymatshenas is a price-comparison app for Iranian shoppers. You ask for something in Persian, a crawl agent running on your own machine queries Iranian marketplaces, and twenty-odd candidate listings come back. Most of them are wrong. Search for a phone and you get phone cases. Search for a fridge and you get someone advertising that they buy fridges rather than sell them. The listings are written by thousands of individual sellers, so a product's name is whatever that seller happened to type.

Something has to look at that pool and decide which rows are actually the thing you asked for, then label each survivor as an exact match, a variant, or merely related. It is the least glamorous call in the system and it is where the product lives or dies. Get it wrong one way and a 250,000-toman screen protector is presented as the cheapest offer on a 105-million-toman phone. Get it wrong the other way and a real product vanishes from the results with no trace.

Nine days ago we moved that call onto a free model from Poolside. This is every number from the log, including the two things it gets wrong.

151 / 151

production calls answered

90 / 90

verdicts parsed first time

2.5s

median — the fastest we measured

27%

of answers leaked our field names

How we measured

Every figure here comes from one of two places, and neither involves writing a new prompt for the occasion. Offline, we keep a corpus of 40 candidate pools that our production chain filtered down to nothing — the hardest cases we have, where the model looked at twenty real listings and kept zero. A harness replays each pool through a candidate using the prompt we actually ship, imported from our build output rather than copied, because scoring a model against a prompt you do not ship measures a prompt you do not ship.

In production, every upstream attempt writes a row: provider, model, latency, tokens, reasoning tokens, finish reason, and which pipeline stage asked for it. Answers are then re-scored by the same validator that runs live, so “how good is this model” and “what would we have refused” are the same question rather than two.

One note on how to read the recovery figures below: they are counts, not rates. A rate whose denominator shrinks when the provider fails rewards failing to answer. Two of the models here lost 4 of their 40 calls to their own transport and then scored 97% of the 36 they answered. Laguna answered all 40.

The comparison

Laguna does not top the recovery chart. It won on the combination: near-best recovery, the lowest latency of anything we tried, and the only model that answered every single call on every run. For a filter sitting inside a search a person is waiting on, that is the trade we want.

What held up

Availability first, because it is not a small claim. 151 production calls on the direct endpoint, 151 served — no 5xx, no timeout, no dropped socket, no empty reply, across all four stages we route through it. We care about this more than most, because the free-model tier churns violently: the endpoint that previously led this chain was measured at 0% available, with 29 of 30 calls rate-limited upstream. The rule we now use to predict it is that a free endpoint served by the model's own vendor holds up, and one served from a shared reseller pool does not.

Second, it does not think when it should not. Our filter is a classification with a strict output contract, and any tokens a model spends reasoning are charged against the same budget as its answer. We have been badly burned by this before — one model lost 95% of its replies to a budget that was too small, returning empty text with a success status for months before anyone noticed. Across 184 Laguna calls, exactly two used any reasoning at all. On the direct endpoint it was one call in 151, which spent 120 tokens and finished cleanly. Its own sibling model averages 998 reasoning tokens per call on the same corpus.

Third, it writes clean Persian. Our writer stage turns the filtered rows into the prose you read, and we score it with a validator that checks three things a model can get catastrophically wrong: quoting prices in the wrong unit (toman and rial differ by a factor of ten), leaking foreign scripts into Persian, and stating figures with no rows behind them. Laguna passed 26 of 26. The paid model it displaced passed 54 of 56, its two failures being one currency mislabel and one Hebrew word dropped into a Persian answer.

Reading the answers by hand, the judgement is good in ways the validator cannot see. Asked for a 110-amp battery, it opened with «در فروشگاه‌های پیگیری‌شده هیچ باتری دقیقاً ۱۱۰ آمپری موجود نیست» — no battery of exactly 110 amps is available in the shops we tracked — and explained that every result was in fact 100 Ah, before quoting a single price.

Where it fell short

Both of these are instruction-following problems in a non-English language, and both were invisible to every automated check we run. We found them by reading the output, which is the only reason we can report them at all.

The first: it prints our internal machinery to customers. Each row we hand the writer carries a machine tag — match=exact, priceFreshness=old. Those are inputs, not words a shopper should ever see. Laguna echoes them into finished Persian sentences in 7 of 26 answers. The model it replaced does this in 0 of 56.

The second is subtler and more interesting. We ask for each kept row to be labelled exact, variant, or related. Laguna sits on the middle rung almost regardless of the evidence.

The clearest single case: a user searched «کیس کامپیوتر» — computer case. All 24 results were literal computer cases. GPT-5.6-luna, Gemini and DeepSeek all called them exact matches. Laguna called 22 of the 24 merely related.

That is not cosmetic, because the label is an instruction to the next stage: when a row is only related, our writer is told to say plainly that it is not quite what the user described. So an under-confident label becomes an answer that apologises for products that were exactly right — on one air-conditioner search it advised the user to «حتماً قبل از خرید با فروشنده تأیید کنید», to confirm with the seller before buying, about rows that were the correct model.

There is a third, milder thing worth writing down: occasional Persian misspellings, «متفاوط» for «متفاوت», and colloquial forms mixed into otherwise formal prose. Nobody has complained. It is the kind of thing a native reader notices and a benchmark does not.

And one thing we suspected and cleared it of. We flag implausible prices as outliers — a bag of cement listed at 4,805 toman when its peers are 230,000. Laguna named such a row as the cheapest in one of its two opportunities, which looked like a defect until we checked the others: the model it replaced does it three times out of three. That is a gap in our prompt, not in this model, and we are fixing it on our side.

Going direct beat going through a reseller

We reached the same model two ways in parallel — Poolside's API directly, and the free variant offered through OpenRouter. Same weights, same prompt, same corpus, same session. The direct endpoint won on every axis: 40 of 40 answered against 36, a 2.5-second median against 4.4, and a 95th percentile of 7.3 seconds against 13.1.

In production the gap was sharper. One call routed through the reseller spent 106.7 seconds and its entire output budget on reasoning, then returned truncated with a success status — exactly the silent failure we build guards against. The direct endpoint produced nothing like it in nine days. An earlier draft of this post credited the model with zero reasoning tokens; checking before publishing turned up those two calls, and the interesting part was that the bad one was not on Poolside's infrastructure. If you are evaluating a model that offers both paths, measure both: they are not the same product.

Why not the others

Where it sits now

Laguna-s-2.1 is the first model tried on three of our five lists — filtering search results, deciding what a question needs, and writing the answer — and second on the other two. No other model in our system leads more than one, and those three are the lists a user waits on.

We have also added Poolside's paid model as the floor beneath the free tier, ahead of the model that previously held that slot. When our free capacity runs out, the fallback is now the same weights at roughly a third of the cost.

The honest constraint is that our first choice is a free preview with no published limit. We have probed its ceiling exactly once rather than repeatedly, because hammering a free endpoint to find its edge is a bad way to treat one. That means we are routing our most important call at a service whose terms could change without notice, and the way we have designed around it is to keep a paid path warm rather than to pretend the risk is not there.

If capacity were not the constraint, three things would change on our side:

• Batch runs. A user uploads a spreadsheet of 150 products and each one needs its own filtering call — roughly 190 calls in a burst, and the single workload our current arrangement cannot absorb.
• A second opinion on hard pools. Running two verdicts and comparing them would let us catch the labelling problem above automatically, instead of by reading output for an afternoon.
• More reports like this one. Every number here exists because we could afford to replay a corpus. Persian-language failure modes are not, in our experience, well covered by public benchmarks.

Would we recommend it?

Yes, with a paid path kept warm behind it. Filter the internal vocabulary out of the prose yourself, do not treat the “related” label as calibrated, and measure your end-to-end response time rather than your per-call latency — every per-call number looked fine on an afternoon we accidentally doubled our own. Under those conditions it has been quietly, unremarkably excellent for nine days, and it is now the first model we reach for when you ask us what something costs.