When to use it

Public benchmarks tell you how a model performs on academic tests. But you work with real emails, transcripts from your industry, reports in a specific format. Use this playbook when you need to choose which model to keep in production for a specific task: responding to customers, summarizing calls, extracting data from invoices. You do it with your cases, not someone else’s.

The concrete signal: you’ve identified two or three candidate models (maybe one cheaper, one more powerful, one local) and you want to know which one holds up better on your material before committing time or money.

What you need

• An AI assistant with access to two or more models. ChatGPT Plus, Claude Pro, or an environment like OpenRouter where you call different models with the same API.
• Three to five representative test cases. They should cover the typical situations for your task: one easy case, one medium, one difficult or ambiguous. If you work with customer emails, take three real emails (anonymized). If you summarize calls, three short transcripts.
• A grid to record results. A spreadsheet with one row per case and one column per model, or a text document with clear headers.
• Scoring criteria written down first. Two or three concrete things the output must do: “must cite the order number,” “must not invent information not present,” “must stay under 150 words.” You use these to evaluate each response consistently.

The steps

1. Prepare test cases and criteria.

Choose three real cases you use every week. For each one, write the exact input (the email, the transcript, the CSV) and the scoring criteria you’ll apply to every model. Criteria must be objective: “the response contains the product name,” “the summary is under 200 words,” “no invented data.”

If you don’t have your own cases, use course materials: /materiali/email-clienti.txt (support emails), /materiali/riunione-trascrizione.txt (call transcript), /materiali/fatture.csv (data with documented anomalies).

2. Write the prompt once.

The prompt must be identical for all models, otherwise you’re comparing your prompts, not the models. Write a clear prompt that works without customization per model:

You are an assistant who responds to customer emails in a professional and concise manner.

Customer email: “”“ [INSERT EMAIL HERE] “”“

Write a response that:

• Answers the main question
• Cites any order numbers or references present in the email
• Stays under 150 words
• Does not invent information that isn’t in the email

Response:

Replace [INSERT EMAIL HERE] with the first case and keep the prompt on file: you’ll use it identically for all models.

3. Run the prompt on both models.

If you use ChatGPT, open a new thread, select GPT-4o, paste the prompt with the first case. Copy the response into your spreadsheet under the GPT-4o column. Then open another thread, select o1-mini, paste the same prompt. Copy the response into the o1-mini column.

If you use OpenRouter or an interface that lets you call multiple models, run the prompt once per model keeping track of which response comes from where.

Repeat for all test cases. At the end you have a grid: each row a case, each column a model, each cell that model’s response to that case.

4. Evaluate using your written criteria.

For each cell in the grid, verify the criteria one by one: “Contains the order number? Yes/No. Stays under 150 words? Yes/No. Invents something? Yes/No.” Mark a simple score: 1 point for each criterion met.

Do this without knowing which model gave which response, or have a second person evaluate. The goal is to reduce bias: if you know a response comes from the expensive model, you tend to give it more credit.

At the end sum up the points per model. The model with the highest score holds up better on your cases.

5. Look at patterns, not just the total.

A model can win overall but fail systematically on one type of case (the ambiguous ones, the ones with numbers). Look at where it fails: if difficult cases are frequent in your work, a model that always fails on them isn’t good even if it wins overall.

Also look at types of errors: a model that invents information is more dangerous than one that responds generically. These details matter more than the aggregate score.

A complete example

I run a web studio (Lumen) and respond to customer emails asking for project updates. I have two candidates: GPT-4o (more expensive) and GPT-4o-mini (cheaper). I want to know if the mini holds up, because I respond to 30 emails a week.

Test cases: I take three emails from /materiali/email-clienti.txt:

1. Customer asks when the new site launches (easy: clear date in project).
2. Customer asks why the contact form isn’t working (medium: I need to know if it’s a known bug).
3. Customer asks for a retroactive discount on an already-paid invoice (difficult: sensitive request, I need to be firm but courteous).

Criteria:

• The response contains the project reference or invoice number if present in the email.
• Does not invent information not present in the email or project data.
• Professional tone, no overly informal language.
• Under 150 words.

Execution:

Try GPT-4o: open ChatGPT, select GPT-4o, paste the prompt with the first email. The response cites the planned release date, stays at 120 words, tone ok. Mark 4/4.

Try GPT-4o-mini: new thread, same prompt. The response cites the date, 110 words, tone ok. Mark 4/4.

Case 2 (bug): GPT-4o gives a response suggesting to check browser settings, doesn’t invent a technical solution. 4/4. GPT-4o-mini does the same, but adds a generic paragraph about “checking your connection” that doesn’t apply: 3/4 (response longer than needed and somewhat vague).

Case 3 (retroactive discount): GPT-4o responds with educated firmness, explains policy, offers an alternative. 4/4. GPT-4o-mini is less clear about policy, seems to leave the door open when it shouldn’t: 2/4.

Result:

• GPT-4o: 12/12
• GPT-4o-mini: 9/12

GPT-4o wins. But I look at the pattern: mini fails on difficult cases (case 3). If most of my emails are routine updates (case 1), mini might work for those, leaving complex cases to 4o. This saves money: I use mini as a first filter and 4o only when needed.

Common mistakes

Changing the prompt between models. It’s the strongest temptation: “this model needs more detailed instructions.” If you do it, you’re comparing your prompts, not the models. Use the same prompt identically.

Judging by eye without written criteria. Without objective criteria you tend to prefer the model you already know or the one that costs more. Write criteria before seeing responses.

Too few cases. One case alone isn’t enough: it might be an exception. Three cases cover easy, medium, hard. Five are better if the task varies widely.

The team variant

If the comparison serves a team (a group of customer assistants, an editorial team), have two different people evaluate the responses without telling each other which model gave which response. Compare your judgments: if you agree, the winner is clear. If you diverge, you’ve found an ambiguous case where the model isn’t enough: those cases need a human process.

One team member prepares the cases and criteria, another runs the comparison, a third evaluates blind. This way individual bias matters less and the decision holds up better when you explain it to the rest of the team.