Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

Authors: Rubin Wei, Jiaqi Cao, Jiarui Wang, Junming Zhang, Qipeng Guo, Bowen Zhou, Zhouhan Lin

arXiv ID: 2607.27919

Problem: Decoder-only language models entangle long-term memory and reasoning in a single parameter set, so memory capacity cannot be scaled independently of the model. The Memory Decoder design (a parametric long-term memory module) had only been shown at small scale; scaling it up is assumed to be infeasible because standard indexing and search pipelines cannot handle a large pretrained memory.

Key Methodology:

Scales the Memory Decoder up to 6.9B parameters, pretrained on 300B tokens
Replaces the standard Faiss pipeline (infeasible at this data scale) with a distributed pipeline for Faiss indexing and retrieval plus sparse, batch-wise loading of kNN distributions
Evaluates parameter-allocation tradeoffs: memory parameters vs base-model parameters, across model scales
Tests both a general 6.9B memory with Pythia bases and 1.7B domain-specific memories with Qwen3 Base models from 0.6B to 14B

Key Results:

Pairing a 6.9B general memory with Pythia-410M raises its 17-benchmark average from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters
1.7B domain memories improve Qwen3 Base models by more than 9 points on average across the three domains, at every scale from 0.6B to 14B
Allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone, across all model scales tested

What it means for developers: Memory can be bought separately from reasoning: a small base model plus a large pretrained memory checkpoint can outdo a much larger monolithic model, which is a direct deployment lever for agents that need broad domain knowledge without a big model budget. The distributed indexing work also shows the operational cost of parametric memory at scale: retrieval infrastructure, not just weights, is part of the bill. Watch for pretrained memory checkpoints released as standalone artifacts.

Paper: arXiv:2607.27919