Three papers published on the same day on Hugging Face Daily Papers converge on a question that benchmarks have so far skipped: does accumulated experience really make agents better over time?

PAST-Bench, ContinualSkillBench, and “When Agents Learn to Be You” measure three facets of the same problem. The first checks whether an agent that preserves preferences, histories, and abilities across sessions improves on subsequent tasks. The second measures whether agents can evolve their skills by reusing them in chained tasks. The third documents the risk that “persona skills”: personal abilities distilled from interaction history, become a vector for private data leakage.

The common answer is honest and disappointing: improvements exist, but are uneven and fragile. PAST-Bench, across seven models and four frameworks, finds that two agents with identical final gains can have achieved them through completely different paths, and not always through the intended pathway of save, retrieve, and update. ContinualSkillBench shows that much of the improvement comes from adaptation to previous context, not from true reusable skills. Less capable models accumulate fragmented collections of domain-specific skills instead of consolidating them.

For those using or building agents in their own work, the implication is concrete. As we discussed on August 4, agent memory costs tokens and must be designed carefully. Now it emerges that even when you design it and pay for it, the return is not guaranteed. An agent can retain information without thereby becoming an agent that learns.

In detail

The three benchmarks fill a gap that agent research had left open. Until now, evaluation has focused on single execution: an agent receives a task, completes it, gets measured on the result. What happens between one task and the next, and especially whether accumulated experience produces a systematically better agent, remained off the radar.

PAST-Bench (Princeton University) builds 26 scenarios and 204 episodes to isolate this variable. The idea is simple but rigorous: each agent moves through ordered task sequences in fresh sessions, and researchers turn accumulated memory on and off to measure the difference. They test seven base models and four agentic frameworks. The result is that improvement exists, but is uneven. Two agents with identical final gains can differ profoundly in the path that led them there: one follows the intended pathway (saves experience, retrieves it when needed, updates it), the other achieves the same result without traversing that path, meaning the gain is accidental, not systematic.

The Hermes+ framework, developed by the same authors, intervenes at five points in the agent loop to enforce the correct path. Average gain increases and pathway traces become clearer, especially in tasks requiring replacement of outdated information. The effect remains dependent on model and capacity, however.

ContinualSkillBench (Peking University) approaches the problem from the skills angle. Five domains, one hundred subtasks each, ordered by increasing difficulty and with cross-reuse opportunities. The main finding is that in-context learning (the model adapts to previous context) produces results comparable to explicit skill maintenance. Much of the improvement, in other words, comes from simple adaptation to previous context, not from true reusable abstractions. Explicit skills help, but only in tasks requiring repeatable procedures or precise outputs. And less capable models tend to accumulate large, fragmented collections of domain-specific skills instead of consolidating them into something transferable.

The third paper, “When Agents Learn to Be You”, brings the discussion to the security front. “Persona skills” are portable artifacts that distill an individual user’s interaction history: preferences, habits, routines. When these skills are reused by downstream agents, they concentrate fragmented personal signals and amplify them. The benchmark, called AntiSkillBench, evaluates leakage and impersonation risks across the entire pipeline. Defenses designed for single records or retrieval-based memory don’t hold up when personal information gets distilled into an executable, portable skill.

The convergence of the three papers says one thing clearly: agent evolution over time is the next bottleneck. The agent that remembers has taken half a step. It must consolidate what it remembers into transferable skills, and do so without turning the user’s personal history into an attack surface. The benchmarks say we’re not there yet, and they say it with numbers.