Hetzner, the German operator known for cheap servers, is experimenting with an LLM inference service with OpenAI-compatible API. For now there’s a single model (Qwen 3.6-35B, a MoE with 3 billion active parameters), no billing, no SLA: it’s a declared experiment to understand if people actually want to use it.

Why it matters to you. AI compute is exiting the lab monopoly. In just days we’ve covered AMD investing 5 billion in Anthropic and Google putting 40 million tokens into research. Hetzner is a different kind of operator: it doesn’t build models, it has no labs, it buys hardware and runs it on razor-thin margins. If open-weight inference becomes commodity, anyone who knows how to buy and operate GPUs cheaply has a real advantage.

Early tests report 153 ms to first token and 224 tokens per second output (July 23, 2026, source: Sliplane). That’s a single measurement, from one client, with no concurrent load. The model follows format instructions but flubs two simple arithmetic calculations: the tier you’d expect from a 35B.

The signal matters more than the numbers. If infrastructure operators like Hetzner enter the inference market, cost per token for open models could drop further. And choosing a provider becomes a question of price and latency, not model access.

In detail

What came before.

Until now, on-demand LLM inference was the domain of three types of players: labs building models (OpenAI, Anthropic, Google), big clouds (AWS Bedrock, Azure) and a galaxy of specialist providers like Together AI, Fireworks or Groq. Hetzner doesn’t fit any of these buckets: it’s a hosting provider selling physical and virtual servers at aggressive prices, with a reputation built on operational efficiency.

What changes.

Hetzner’s experiment is small but hits a weak spot in today’s market. Open-weight models, once downloaded, are identical for everyone: anyone can grab the same Qwen weights, run them with the same serving software (vLLM, TGI) and expose them via OpenAI-compatible API. This means differentiation is hard and margins compress quickly.

Competitive advantage, then, shifts to three things: buy GPUs cheap, keep them busy, and run data centers at low cost. Hetzner built its entire company on exactly these three things.

There’s also a capacity story. A bare-metal GPU server rented to a customer is theirs, used or not. An inference API shares the same GPU across many users and keeps it busy. If Hetzner has spare GPU capacity, or plans to scale it, an inference product turns that fixed cost into revenue.

Technical details (no jargon).

The API is compatible with the standard OpenAI client: change the base_url, put in your token and it works. The model served is Qwen 3.6-35B-A3B-FP8, a Mixture-of-Experts architecture: 35 billion total parameters but only 3 billion active per generated token, because the model activates only the “experts” relevant to each text snippet. Weights are quantized in FP8, an 8-bit format that cuts memory needs in half versus classic FP16. Files weigh around 38 GB, fitting comfortably in the 96 GB VRAM GPUs that Hetzner has in its servers.

One interesting detail: the model has an enable_thinking option that, if not explicitly disabled, burns a surprising chunk of completion budget on invisible reasoning before returning the answer. It works in tests but isn’t documented by Hetzner.

How much to trust this.

Not much, for now, and that’s fine. Hetzner makes it clear: no billing, no SLA, no production guarantee. A single model, a single documented GPU type, no information on how it scales under load. Latency numbers (153 ms to first token, 224 tokens/s output) come from a single test with one client and tell us almost nothing about behavior with many concurrent users.

The hardware also raises an open question. GPUs published in Hetzner’s catalog are NVIDIA RTX 4000 SFF Ada (20 GB) and RTX PRO 6000 Blackwell Max-Q (96 GB). These are workstation GPUs, not the H100 or B200 that specialist providers use. Fine for small and medium models, but not for frontier ones.

What not to conclude.

This doesn’t mean Hetzner replaces OpenAI or Anthropic. It means the open-weight inference market, today fragmented across niche providers, is attracting operators with different scale and costs. If the trend holds, users of open models will have more options and lower prices. Anyone dependent on proprietary frontier models sees no immediate change.