Agent memory works better when you stop retrieving every memory the same way

Most agent memory systems embed every past conversation turn and search them all with one retrieval trick. A new paper from City University of Hong Kong and Huawei Noah’s Ark Lab argues that this is the bottleneck: “what happened last Tuesday,” “I’m allergic to peanuts,” and “how does RAID 5 work” are different kinds of memory and need different search strategies. Their system, MemoType, routes each memory and query to one of three types and retrieves accordingly. On the hardest benchmark, Recall@1 jumped from 31.91 to 48.09.

• Best Recall@1 on all four benchmarks against nine baselines, including HippoRAG2, A-Mem, MemoryOS, and LightMem
• +16.18 points on LongMemEval-M (48.09 vs. 31.91 for SeCom), the largest gain in the paper
• About 5 points from typing alone: routed retrieval beat the best single strategy by 5.14 to 5.68 Recall@1 points on three benchmarks
• $3.77 vs. $548.98 to build memory for LongMemEval-M, roughly 146x cheaper than HippoRAG2 and 248x cheaper than A-Mem
• A 1.7B router classified memory types at 92.10% accuracy, ahead of GPT-4o-mini at 83.55%

Three kinds of memory, three retrieval strategies

The taxonomy comes from cognitive psychology (Tulving’s episodic vs. semantic split), with semantic memory divided in two:

• Episodic (EM): events tied to a time, place, and people. “We moved the launch to Friday after the outage.”
• Personal semantic (PM): stable facts about the user. “I’m vegetarian.” “My team uses Go.”
• General semantic (GM): world knowledge that came up in conversation. “Explain how B-trees split.”

A single memory can carry more than one label. A LoRA-tuned Qwen3-1.7B labels each turn at ingestion time. At query time, the system asks an LLM to write five hypothetical memories that would answer the query, classifies each one, and takes the majority type. Filtering to that type alone removes about half of the unrelated memories before any similarity search runs.

The intuition holds up once you look at the strategies. A query like “what did we decide in the Thursday meeting with Sam?” is mostly matching on time and participants, so breaking the memory into those fields helps. A personal fact like “I’m vegetarian” is phrased nothing like the question that needs it (“what should I cook for dinner?”), so indexing the questions a memory answers works better than indexing the memory itself. Each trick helps its own type and can hurt the others, which is why no single strategy wins everywhere.

The authors also give a theoretical argument: when a corpus contains clusters of genuinely different content, any one query embedding has a ceiling on how well it can align with all of them. That proof assumes uniform, well-separated categories, so treat it as motivation more than a guarantee.

The results

Every method used the same Contriever retriever and gpt-4o-mini as the answer model, so the comparison isolates how memories are stored and searched.

The gains are lopsided. They are large on LongMemEval-M, a haystack with far more sessions per user, and on PerLTQA, which is dominated by personal facts and events. On LongMemEval-S and LoCoMo the margin is only 2 to 2.3 points. That pattern fits the thesis: type filtering matters most when the pile is large and the wrong type of memory crowds out the right one.

The gains carried through to answers. GPT-judged QA accuracy went from 36.20 to 50.20 on LongMemEval-M, 47.00 to 56.80 on LongMemEval-S, and 40.18 to 48.79 on LoCoMo. PerLTQA moved less, from 60.11 to 62.83.

The cleanest evidence for the typing idea is the ablation. Holding everything else fixed, routed retrieval beat the best single strategy by 5.24 points on LongMemEval-S, 5.14 on LoCoMo, and 5.68 on PerLTQA. The best single strategy was different on each benchmark, which is the paper’s point in miniature. The final LLM pruning step, by contrast, barely matters: it added 0.76 F1 on LongMemEval-S.

It is also much cheaper to build

Graph and note-based memory systems like HippoRAG2 and A-Mem call an LLM on nearly every turn to extract entities or write linked notes. MemoType’s ingestion is a small classifier plus LLM work on the subset that needs it: element extraction for episodic turns and question generation for personal ones. On LongMemEval-M, only 2.2% of turns were episodic.

The cost gap explains a missing column: A-Mem, HippoRAG2, and RAPTOR have no LongMemEval-M retrieval numbers because building their memory took more than a week.

For practitioners, the router is the most reusable piece. A fine-tuned 1.7B model hit 92.10% classification accuracy on the authors’ 6,000-sample TriMEM dataset, against 83.55% for prompted GPT-4o-mini and 77.75% for Qwen3-32B. It also transferred to PerLTQA at 96 to 100% accuracy without training on it. A cheap classifier in front of your vector store is an easy thing to add to an existing RAG stack.

Caveats

• Query-time cost is not reported. The cost table covers ingestion only. Every query still triggers hypothetical-memory generation and an LLM pruning pass, and the paper gives no latency numbers.
• Half the benchmarks show small gains. The 16-point headline comes from LongMemEval-M. On LoCoMo and LongMemEval-S, MemoType is ahead by roughly 2 points.
• The router is trained on in-house labels. TriMEM was labeled by GPT-5.2 and spot-checked by three annotators on 20% of samples (Fleiss’ κ = 0.87). The paper also lists source counts that sum to 3,000, not the stated 6,000, without explanation.
• Prompted LLMs were better at one type. GPT-4o-mini beat the trained router on personal semantic accuracy (87.18 vs. 82.65), the most common type in the dataset.
• Different models answer and judge. Answers come from gpt-4o-mini; GPT-5.2 grades them. Judge-based QA scores should be read alongside the F1 numbers, which show the same direction.
• Version 1 preprint. Code is on GitHub, but the results have not been independently reproduced.

The broader lesson is a familiar one from search engineering: one ranking function for a heterogeneous corpus leaves accuracy on the table. Agent memory is heterogeneous by nature, mixing events, user facts, and reference material in the same chat log. Classifying before you retrieve is cheap, and in this paper it was worth 5 points of Recall@1 on its own.