A week of Hetzner's free inference in production

A week ago we published a benchmark of Hetzner's two free Qwen models, replayed over 112 recorded Persian searches. A benchmark tells you what a model can do under laboratory conditions. It does not tell you whether the thing stays up, answers on a Tuesday afternoon, or survives contact with a real user waiting for a price. So we shipped it, logged every single call, and this is the week that followed.

27 / 27

replies were valid JSON

0 / 8

healthy searches came back empty

4.1s

median call, reasoning off

$0

billed for any of it

What we are measuring, and how big the sample is

Every request our backend makes to any language model is written to a log row with its model, latency, token counts and outcome. Nothing here is reconstructed after the fact and nothing is a replay — these are the calls that answered real people searching for real products.

We are not going to dress up the sample size. Since we settled on the current configuration on 20 August, eleven production calls have gone to Hetzner, and twenty-seven verdicts fall inside our seven-day evidence window. That is a small number and the reason it is small is the whole point of the last section of this post. Read the reliability figures as “nothing has gone wrong yet”, not as “nothing can”.

The week in production

The quality side matters more than the latency side, because a filter that answers quickly and wrongly is worse than one that is slow. Over the same window: every one of the 27 replies was parseable JSON, none of the 8 healthy candidate pools came back empty, none kept the entire pool, and the models kept 44% and 50% of the candidates they were shown — which is what a filter that is actually filtering looks like. Not one reply hit the output ceiling. Ten of ten trials passed our output validator with zero violations.

Reasoning is still the whole game

The single biggest lever is not the model, the prompt or the hardware. It is whether you let these models think before answering. We already knew that from the benchmark; production put a price on it.

What we got wrong, in public

On 19 August we promoted Hetzner to the front of our chain with reasoning enabled on both models, on the theory that better verdicts were worth the wait. Four hours of live traffic priced that theory and the answer was no. The median time to answer a price question across the entire product went from 56 seconds to 115 seconds, and the 90th percentile from 136 seconds to 292. Twenty-two searches spent 2,542 seconds sitting in the relevance filter; the three searches in the same window that fell through to our paid model spent eleven seconds between them. Two people pressed Stop on searches whose prices had already been fetched and were sitting there, waiting on a filter that was still thinking.

We reverted the same afternoon. It is worth being precise about whose fault that was: it was ours. The endpoint did exactly what we asked it to. We asked for the expensive configuration.

The model we wrote off has quietly started working

In the benchmark, the dense Qwen3.8-27B with reasoning off was the worst entry on the board. Its median output was six tokens: it replied with an empty result and stopped. Fast, clean, and useless — we scored it 9 out of 100 and moved on.

The path we use to trial models always disables reasoning. So every call the dense model has served since 20 August has used the exact configuration we had written off — and it is now returning complete verdicts of 130 to 270 tokens, answering all six times it was asked, in a median of 6.9 seconds, without blanking a single one of the five healthy searches it saw.

We cannot fully separate how much of that is Hetzner and how much is us from the outside, and it would be dishonest to claim otherwise on six calls. But our side of the configuration has not changed between the two measurements, and the behaviour has. Something on the serving side is doing better than it was a week ago. If you tried this model with reasoning off and gave up, it is worth another look.

The one thing still holding us back is not the model

Our key allows ten requests per sixty seconds, shared across our entire backend. In the whole life of this integration we have never once been rate-limited — not a single rejection across every call we have made.

That is not a compliment to the ceiling. It is a description of how little traffic we dare send through it. One user question costs us several calls, and a bulk price run over a 150-item shopping list is roughly 190 of them, which at ten a minute is nineteen minutes of queueing before any generation starts. So the endpoint sits second in our chain, behind a paid model, and only serves when the paid one is cooling down. That is why this post reports eleven calls instead of eleven thousand.

It is the ceiling, not the quality, that decides how much of our workload can live here. Our backend already runs on a Hetzner cloud server in Finland, so raising it would simply move traffic from a paid API into Hetzner's own network. We would take that trade immediately.

Would we recommend it?

Yes, with the configuration written down. Turn reasoning off, keep a short per-call deadline, put a paid model behind it for whatever it misses, and do not promote it to primary without measuring what happens to your end-to-end latency rather than your per-call latency — every per-call number looked fine on the afternoon we doubled our own response time. Under those conditions it has been quietly, unremarkably reliable for a week, and it has cost nothing. That combination is rarer than it sounds, and it is now part of how we compare prices.