CrucibleBench puts language models inside a MUD, a 90s-style text world, and evaluates them over 50 turns with hidden social objectives. Thirteen models, 650 runs, $99 in total API costs. The main finding concerns how you measure: the LLM judge used for scoring shuffled the rankings by up to six positions, while aggregate reliability statistics stayed silent.

For anyone building with AI, the point is constraints. A MUD has a finite action space (7 command types, 12 rooms, 14 objects): hallucinations surface immediately, because the model can’t invent an action that doesn’t exist. NPCs have trust and suspicion states (0-100) that shift through dialogue: social feedback is explicit, not interpreted after the fact. This is the pattern we saw in the music clip benchmark: useful tests emerge from real problems with clear constraints.

The second warning concerns anyone using an LLM as a judge. Per-model agreement between the classifier and an independent judge ranges from 21.7% to 84.8%, but the aggregate statistic (κ = 0.04) doesn’t show it. If you use one model to evaluate another’s responses, check agreement on individual cases, not just the average.

All 650 transcripts are public at cruciblebench.ai along with the whitepaper, if you want to read the raw run logs.

In detail

The idea behind CrucibleBench borrows a philosophy Nintendo attributed to Gunpei Yokoi: mature, affordable, well-understood technology used in a new way. MUDs (Multi-User Dungeons) are text-based worlds that have existed since the 80s. No graphics, no photorealistic simulation: text commands, rooms described in prose, objects you pick up and drop.

These constraints are the point. In a standard benchmark, a model answers isolated questions. In a MUD, it must manage persistent state: an object picked up stays picked up, trust earned stays earned. If the model repeats the same failed approach with an NPC, the game logs it. If it tries talking to someone who isn’t there, the room responds “there’s no one here.” Every run leaves a complete, readable transcript.

The three failure modes the benchmark detects are familiar to anyone who’s run agents in production. Dialogue looping (14-66% of frontier model runs) is the agent repeating the same phrase to an NPC that won’t respond as hoped: the text cousin of a support bot spinning in circles. Wrong-room interaction is the model talking to someone who isn’t there, because it lost track of its location: the equivalent of calling an API out of scope. Exploration paralysis is the model looking and re-looking without ever acting.

The most interesting result concerns how you measure. The LLM-judge component inside the scoring stack reshuffles the ranking tops by up to six positions. Two of the four evaluated dimensions pass through a dialogue classifier. Per-model agreement between this classifier and an independent judge varies from 21.7% to 84.8%. The aggregate reliability statistic (κ = 0.04) doesn’t reveal this instability. This result applies to anyone using an LLM to evaluate another, and aligns with what we reported on the LLM-as-judge paper: strong models can judge almost like humans, but with known and predictable biases.

The limitations are stated. It’s Phase 1, proof-of-concept. Confidence intervals for the top 8 positions substantially overlap: read the ranking as exploratory, not definitive. Fifty turns per run, temperature 0.3, 5 seeds per model. The $99 budget shows that evaluating agent behavior doesn’t require expensive infrastructure, but this doesn’t mean results generalize beyond the game domain.

For anyone building their own testbed, the methodological lesson is: clear constraints and verifiable objectives beat abstract metrics. The course on how to evaluate without benchmarks follows the same logic: generic benchmarks don’t tell you if your agent works for you.