Alex Wang: every transformer layer is a memory store, and capacity is set by interference

Alex Wang (@heyyalexwang) published a blog post, "You could have invented transformers from memory". It treats each transformer layer as a memory store and asks how much each can hold. The post comes with code and interactive figures.

Recalling the character 記 from an outer-product store, a ReLU MLP, a gated MLP and self-attention. @heyyalexwang

The setup. A memory is a key-value pair. Recalling one means querying with a key, which returns the target value plus interference from every other stored memory with a similar key. Wang defines a store's capacity as the number of memories it can hold before interference grows as large as the signal.

The progression, per the post:

• Outer-product (linear) memory. The store accumulates v kᵀ terms. With random unit keys, interference-to-signal scales as roughly √(M/d), so capacity is about d.
• Optimal linear store. Solving the least-squares problem whitens the keys. It recalls exactly up to M = d, and its capacity is about 2d.
• MLPs. Passing each key similarity through a nonlinearity suppresses weakly matching keys, which gives a two-layer MLP. ReLU is the thresholded version. Squared ReLU is a GLU-style MLP. Raising similarities to the power p gives a capacity of roughly d^p. The thread summarizes this as ReLU giving roughly O(d²) with squaring, and d³ with cubing.
• Self-attention. Since the exponential is a sum of powers, softmax attention follows. With d = 64 keys and √d rescaling, Wang estimates it can store over a million pairs. He says more is possible with log M rescaling for long context. The post puts the progression at about 64 memories for a linear layer, about 2,700 for an MLP, and over a million for attention.

Linear memories with d = 64: the outer product fails near M = 64; the least-squares store is exact up to M = d. Alex Wang

Sharper nonlinearities push the failure point right: ReLU², SwiGLU and ReLU³ hold far more memories than a linear score. Alex Wang

Adding terms of the exponential series lowers interference; the full exponential (self-attention) stays far below the failure line. Alex Wang

Caveats from the post and replies.

• The estimates assume random keys and values and are back-of-the-envelope. Wang agreed that models trained on real data will behave differently.
• Replies noted that capacity for retrieval differs from bytes stored, and that correlated keys, such as near-duplicate facts, change the picture. One reply, from the authors of a related study, reported that name-like keys cause far more forgetting than random IDs.
• Others pointed to existing work on Hopfield networks and on kernel-regression views of sparse attention. Wang's reply was that he was "just showing people something cool".
• Another reply argued that single-layer analyses may not transfer to multi-layer models, and that learning is about generalization rather than memorization.

Related work. Wang points to prior work on feedforward layers as memories (Sukhbaatar et al.; Geva et al.). A recent paper on associative-memory views of transformers, and one on fine-tuning-induced hallucination, both frame forgetting as interference between overlapping facts.

#Why it matters

The post gives a compact, checkable intuition for why MLPs and attention look the way they do. It frames memory capacity as a question of suppressing interference. That view also bears on interpretability and on why adding new facts can degrade old ones.