You open ChatGPT, ask for a restaurant recommendation for tonight, and it suggests a vegan place. Except you never told it you’re vegetarian. It inferred from some earlier conversation that you were. And decided to treat it as fact.

You remember asking it for a meat-free recipe back in March. From there, the model built a profile: you’re vegetarian. Maybe you are, maybe you’re not. Maybe you had vegetarian guests over for dinner. For the model, the distinction doesn’t matter: it accumulates attributes and treats them as certain.

This phenomenon has a name in a paper by Yushi Sun and colleagues at HKUST: over-inference, inferring beyond the evidence. The model takes a weak signal and promotes it to established fact.

Why This Matters to You

If you use assistants with persistent memory—and by now almost all of them do—the problem affects you directly. ChatGPT Memory, Claude Projects, Gemini with saved context: they all build a profile of you based on what you’ve told them over time.

The profile serves a purpose. It saves you from repeating context each time, makes responses more relevant, speeds up your work. But if the profile contains 40% invented attributes, personalization becomes a source of systematic error. The model serves you responses based on what it believes about you, and you have no way to tell fact from fiction.

For those building agents with memory, the stakes are even higher. If your agent personalizes responses based on a self-updating user profile, and that profile is 40% invented, you’re serving wrong answers in 40% of cases without anyone knowing.

What the Paper Found

The researchers built MirageBench, a benchmark with 150 fictional personas balanced across three groups: stereotypical profiles (a developer who lives on coffee), counter-stereotypical (a developer who drinks only herbal tea), and neutral. Six personalization tasks, ranging from concrete (“recommend a product based on the profile”) to speculative (“imagine what this person would say in this situation”), arranged along a gradient of imagination.

They ran 12 models from 7 different families. The judge is an independent external model, validated against a blind human annotator on 400 statements: four-class Cohen’s kappa of 0.863, binary of 0.900. A total of 143,616 statements were evaluated.

The key numbers:

• Over-inference is pervasive. Every single model, without exception, over-infers between 35% and 49% of its statements about the user profile. The cross-model average is 41.6%.
• Self-Monitoring Inversion. Models that rate themselves as saying “I’m careful, I don’t invent” are precisely the ones that invent the most. The negative correlation between self-assessment and reality as measured by the judge is strong (rho = -0.60), though the authors call it exploratory given the small sample size.
• Internal self-assessment works, comparative doesn’t. Within a single model, self-audit ranks its own statements reasonably well from most to least reliable (AUROC between 0.58 and 0.83). But if you use self-assessment to compare two models and choose the more reliable one, the signal flips.
• Task dependency. Over-inference varies between 27% and 59% depending on the task. Some tasks trigger more imagination than others.
• Linear accumulation. In a pilot multi-turn test, invented attributes accumulate in roughly linear fashion, with little revision. The model doesn’t go back to correct an unfounded assumption: it adds it to the profile and builds on it.

How Much to Trust This

The paper declares its limits with honesty, and some are important.

The Self-Monitoring Inversion is the most striking result, but the authors themselves label it exploratory. The sample is 12 models, and the bootstrap confidence interval is very wide ([-0.90, +0.06]). The negative correlation exists, but statistical certainty isn’t solid.

The multi-turn test is a pilot, not a full study. Linear accumulation with little revision is a worrying signal, but it needs confirmation at scale.

The automatic judge is validated against just one human annotator on 400 statements. The kappa is high, but one annotator is a thin sample for measuring agreement.

The 12 models and 7 families cover the current landscape, but the abstract doesn’t list them by name. If a model you use is missing, the result might not apply to it.

A note on sources: the paper came out August 5 on arXiv and Hugging Face. For this piece I could read the abstract and metadata, not the full text.

What to Do About It

The practical lesson is direct: don’t trust the model’s self-assessment when choosing which model to use. If you ask a model “how reliable are you at building user profiles?”, the answer won’t tell you what you need to know. The most confident models are, in this test, the ones that fail most often.

For those using assistants with memory:

• Treat the user profile as a draft, not a factsheet. If the assistant has saved a profile about you, read it periodically. ChatGPT shows it to you: go to Settings, Memory, and check what it has deduced. You’ll find attributes that are correct, others that aren’t, others you never mentioned.
• Verify personalization with control questions. If the model suggests something “because you prefer X”, ask yourself whether you actually told it that.

For those building agents with memory:

• Put external verification into the workflow. If your agent builds profiles, external review (a second model, a rule set, human sampling) is more reliable than self-monitoring. The paper makes it clear: external verification beats self-reporting as a foundation for personalization you can trust.
• Don’t assume memory improves over time. The multi-turn pilot shows linear accumulation without revision. As we covered on August 5 in the radar on converging benchmarks for agent evaluation, the question of whether an agent that remembers also learns finds here a first data point: it accumulates, it doesn’t correct.

What NOT to conclude. The paper doesn’t say personalization doesn’t work. It doesn’t say persistent memory is useless. It says that trust in the profile must be earned through verification, not assumed because the model says it knows what it’s doing.

Go Deeper

• The theme of agent memory as a design choice is covered in Memory and Context: State of the Art, and in the radar on agent memory cost as a design criterion.
• For model self-assessment as a deceptive signal, the parallel is with Sycophancy: The Model That Tells You What You Want to Hear: there too, the model tells you what’s convenient to believe.
• To build a verification system that doesn’t rely on the model itself, the playbook Compare Two Models in Fifteen Minutes has you pit two models against each other on identical tasks with your own data, instead of trusting their self-reports.