paper
arXiv cs.AI
October 31st, 2026 at 4:00 AM

When Stored Evidence Stops Being Usable: Scale-Conditioned Evaluation of Agent Memory

arXiv:2605.07313v2 Announce Type: replace Abstract: Memory-agent evaluations report fixed-snapshot accuracy or retrieval quality, but these scores do not show whether evidence remains usable as irrelevant sessions (sessions not annotated as task-relevant evidence for the query) accumulate. We present a scale-conditioned evaluation protocol for agent memory under evidence-preserving growth: for each query, task evidence is held fixed while irrelevant sessions are added. The protocol logs agent-memory trajectories and reports four diagnostics: budget-compliant reliability, tail memory-call burden, failure-regime decomposition, and the usable-scale boundary where reliability falls below the target. Applied to LongMemEval and LoCoMo across flat, planar, and hierarchical memory interfaces, the protocol shows reliability loss is not a single phenomenon. On LongMemEval, HippoRAG stays within the two-call budget but loses 16-20 percentage points in budget-compliant reliability as irrelevant sessions are added; LiCoMemory's observed failures depend strongly on the agent, with Qwen3-8B exceeding the budget while Qwen3-32B and Qwen3-235B remain reliable in the tested range. The result supports a framework for making scalable-memory claims conditional on agent, interface, scale range, and interaction budget. Code and evaluation artifacts: https://github.com/AndyLu666/memory-at-scale

#ai
#open_source

Get stories like this every Monday

The week's most important AI news in one email. Free, unsubscribe anytime.

Score: 2.94

Engagement proxy: 0

Canonical link: https://arxiv.org/abs/2605.07313