A language model on your own laptop: can it replace the cloud?
We ran a 27-billion-parameter model on a MacBook and put it against four cloud models, on the same prompt and the same real Persian searches. With reasoning on, the local model matches the best cloud model — but each call takes 165 seconds. With reasoning off it gets 24× faster and its quality drops to that of the weakest model on the list.
Every price search in our product needs a language model at least twice, and both calls happen in the cloud. A model running on the user's own machine would have two clear advantages: the per-call cost goes to zero, and the search text never leaves that machine. So the question is simple: is a local model good enough? This report is the measured answer, on our own real data.
- 27B
- parameters, 8-bit, on a laptop
- 6
- model configurations, one shared prompt
- 53
- real Persian searches from our logs
- 165 s
- per call for the local model
The job: which listing is actually the product?
When someone searches for «واسکازین ۹۰W140» — a gear oil — our crawler comes back with 20 to 24 listings from different shops. Among them are gear oils, engine oils, and sometimes things that are entirely unrelated. The model's job is to keep only the genuine product and drop the rest, as strict JSON with no commentary.
This job fails in two directions and both are bad for the user: the model can be so strict that it keeps nothing — the user sees an empty page even though the product was in the list — or so loose that it keeps everything, which means there is no filter at all. We measured both separately.
Method: the same prompt, the same searches
We took 34 hard searches where our current filter had in fact kept nothing, plus 19 healthy ones to check that models do not simply loosen up. Every model sees exactly the candidate list the live model saw at the time, and the prompt is read from the product's own build rather than retyped — benchmarking a prompt you do not ship measures nothing useful.
The local model is Qwen3.8-27B at 8-bit, served by MLX on the laptop itself, and we measured it in two configurations: with reasoning on and with reasoning off. The four cloud models are the ones that actually appear in the product's provider chain.
Finding one: without reasoning, the local model is the weakest option
- openai/gpt-5.6-lunacloud · 4.2 s97
- inclusionai/ling-3.0-flashcloud · 10.2 s97
- google/gemma-4-31b-itcloud · 1.7 s82
- Qwen3.8-27B on a laptop, no reasoninglocal · 8.8 s21
- gemini-3.5-flash-litecloud · 1.2 s21
With reasoning off, the local model rescued only 7 of the 34 hard searches — the same rate as the weakest cloud model on the list, and less than half of gemma's, which is also free. In most cases its whole answer was six tokens: an empty list. On the healthy searches it also kept the entire candidate list 11% of the time, which is the state in which the filter is effectively switched off.
Finding two: with reasoning it gets good, and very slow
With reasoning on, the same model found the right answer on all five hard searches we ran, with no formatting failures at all. But each call took a median of 165 seconds. For comparison, the best cloud model does the same job in 4.2 seconds. That is roughly 39× slower, for one decision in a chain the user is waiting on.
Read that number with care: we measured the reasoning-on configuration on only 5 searches, because each call takes close to three minutes and a full run over the same 34 would have taken more than an hour and a half. So the quality result is preliminary and we are not calling it settled. What is settled is the speed gap — and the speed gap alone decides the question.
| Model | Empty searches rescued | Per call | Output tokens |
|---|---|---|---|
| openai/gpt-5.6-luna | 97% | 4.2 s | 225 |
| inclusionai/ling-3.0-flash | 97% | 10.2 s | 1,329 |
| google/gemma-4-31b-it | 82% | 1.7 s | 41 |
| gemini-3.5-flash-lite | 21% | 1.2 s | — |
| Qwen3.8-27B local, reasoning on | 100% | 165 s | 1,229 |
| Qwen3.8-27B local, reasoning off | 21% | 8.8 s | 6 |
Why is it so slow?
The cause is not the job itself but how much text the model writes to do it. Of roughly 1,230 output tokens per call, about 960 are reasoning — thinking the user never sees, produced only to arrive at that short final list. On a laptop, generating each token is bound by memory bandwidth, and a dense 27B model has to read all of its weights out of memory for every single token.
There are two ways to improve this and the difference matters. A lighter quantization — 5-bit instead of 8-bit — helps roughly in proportion: expect about 1.6× faster, which turns 165 seconds into about 105 and does not solve the problem. The effective change is architectural: a mixture-of-experts model, which activates only a small fraction of its weights per token, can be an order of magnitude faster. Anyone seriously trying a local model should try that direction, not a smaller quantization.
What this means for us
- For the live search path the answer today is no: someone asking a price question cannot wait three minutes for one step out of several.
- For unhurried batch work — a long list processed overnight, say — 165 seconds is no longer a hard blocker, and that case deserves a proper measurement.
- Turning reasoning off is not the fix: it buys 24× the speed and gives back quality down to the weakest option on the list.
- The real advantage of a local model is privacy and independence, not cost and not quality — and that advantage has to be weighed against exactly this speed gap.
The limits of this benchmark, stated plainly: one machine, one quantization, one local model, and only 5 samples for the reasoning-on configuration. The cloud numbers are over 34 and 19 searches and line up with our previous benchmark, which is good evidence the sample is representative.
This report follows two earlier ones: 15 language models on two real jobs and which model understands Persian product search best.