We benchmarked Hetzner's free Qwen models on 112 real searches

Every price search in our product needs a language model at least twice, and the bill for that is the single largest running cost we have. So when Hetzner opened a free, OpenAI-compatible inference API serving two open Qwen models, the question was not whether it was cheap — it is free — but whether it is good enough to put in front of real traffic. This is what we measured, on our own logged searches, with the prompt we actually ship.

112

real Persian searches replayed

57 / 100

best free score — beats two paid models

$0

cost for the whole benchmark

3.8s

median call, reasoning off

The task: which of these listings is actually the product?

When somebody searches for a specific gear oil, our crawler comes back with roughly 20 to 24 listings from different shops. Mixed in with the right product are engine oils, different viscosities, accessories, and occasionally something entirely unrelated. The model's job is to keep only the genuine product and label each kept row, returning strict JSON and nothing else.

This task fails in two directions and both hurt the user. A model can be so strict that it keeps nothing — the shopper sees an empty result even though the product was in the list — or so loose that it keeps the entire pool, which means there is no filter at all. We measured both separately, and we used the hardest possible sample: the 112 searches where our production filter kept nothing. Recovering those is the thing we actually want.

Method, and why it is reproducible

Each model sees exactly the candidate list the live model saw, and the prompt is imported from the shipped build rather than retyped — benchmarking a prompt you do not ship is a measurement of nothing. Temperature is zero, the output ceiling is 5000 tokens, and every model gets the same two. The score below is simply the percentage of those 112 blanked searches a model rescued; the failure modes that a single percentage would hide are in the table underneath it.

The ranking

The headline is the third bar. A free model beat two models we pay for, including the one currently doing this job in production — and it beat the paid Gemini tier by better than two to one. That is a real result and it is why Hetzner's endpoint is now part of our chain rather than a curiosity.

It is not the best model on the list, and we are not claiming otherwise. Both paid leaders are clearly stronger, and 57 against 100 is a wide gap. What 57 buys is a large share of the volume at no cost, with the paid models still behind it for everything it misses.

Reasoning decided everything — in opposite directions

Both Hetzner models are reasoning models that think before answering, and this turned out to be the least guessable part of the whole exercise. The two models want opposite settings, and getting it wrong is silent in both cases.

The MoE must not think. With reasoning on, one 20-candidate search took 166 seconds and produced 3,697 output tokens, of which all but about 70 were thinking — it spent 164 of those seconds before emitting a single character of its answer. With reasoning off, the same search took 7.6 seconds and 146 tokens, and gave a better answer: it committed to “exact” and “variant” labels where the thinking run hedged everything as merely “related”.

The dense model is the mirror image. With reasoning off it is fast, clean and useless: a median output of six tokens, because it replies with an empty result and stops. That is the worst shape a filter can have, because nothing downstream can tell “I found no match” from “I did not try”. Turn reasoning on and its answers are good — it rescued every hard search we managed to run it on — but it needs 50 to 57 seconds per call, against a hard 60-second ceiling on the connection. Good, and too slow to use.

One more finding: which model is up changes during the day

Within a single afternoon we saw the availability of the two models swap completely. Early on, the dense model returned nothing at all — five attempts, every one dropping the connection at exactly 60 seconds, including for a request whose entire job was to reply with the word “ok” — while the MoE answered in about four seconds. Ninety minutes later the dense model was answering in 5.7 seconds and the MoE was the one timing out.

This is an experimental service and we are not going to pretend that is a scandal; it is the expected cost of free capacity, and it shaped our integration rather than discouraging it. We call both models, fastest-healthy-first, with a short per-model deadline and a back-off that sidelines whichever one is currently unwell. A free leg is a bonus, so if it does not answer quickly we stop waiting and fall through to a paid model — cutting a slow call short actually makes the user's reply faster, and the only thing spent is a free request.

What Hetzner got right

• The API is genuinely OpenAI-compatible. Our provider layer already speaks that dialect, so the integration was one new class and no adapter work — it answered on the first attempt.
• The model choice is serious. These are two capable open models with a 262k context window, not a token demo, and one of them outscored paid models we were already using.
• They state that request and response content is not stored, and that they do not plan to. For a product where the search text is a customer's shopping list, that is a feature, not boilerplate.
• The input token allowance is generous — four million per minute is far more than a request-shaped workload can use.
• It is free during the experiment, with a promise to give notice before that changes. That is enough to justify building against it.

What would help most, as feedback

• Ten requests per minute per key is the binding constraint, not quality. One user question costs us several calls and a bulk price run is around 190, so the ceiling — not the model — is what limits how much of our traffic can land here.
• There is an undocumented ~60 second limit on a non-streaming request: the connection is reset with no status code and no body. Because a reasoning model can legitimately think for longer than that, the failure looks like a network fault rather than a timeout. Documenting it, or returning a 504, would save the next person an afternoon.
• A model can be listed by /v1/models while not actually serving. Having the listing reflect real readiness would let clients route around a cold model instead of discovering it with a 60-second stall.
• Please include reasoning token counts in usage.completion_tokens_details. They are billed as output and they consume the same max_tokens budget, so without them you cannot tell a truncated answer from a bad one — this exact blind spot cost us a real bug once already.
• A documented way to set the reasoning toggle would help. It works today through chat_template_kwargs, but a template that does not understand that field ignores it silently, which turns the single most important setting into a guess.

The short version: we shipped it. Hetzner's free endpoint now handles part of the relevance work behind our price comparison, with the paid chain behind it for what it misses. If you want the harness, both benchmarks are one command each and read from our own recorded searches.