Continuous Memory Machines addresses a bottleneck in recurrent models: short-term computation and longer-term retention often compete within the same state representation.
The architecture builds on Sakana AI’s Continuous Thought Machine. One matrix tracks recent neural activity for computation, while a second stores information for later steps. A Transformer jointly updates the two memories, allowing each store to read from and write to the other.
The paper reports stronger performance and generalization than its evaluated baselines across algorithmic tasks, in-context learning, and recurrent reasoning. Attention patterns expose how the model uses its memory during those tasks.
The contribution is an architecture that separates computation from retention. Its task-suite results do not establish unlimited memory or universal superiority over deployed language models.
Continuous Memory Machines — paperSakana AI — codeDAIR.AI — supplied source post
