@heyyalexwang

Alex Wang: every transformer layer is a memory store, and capacity is set by interference

Alex Wang's new blog post derives the linear layer, MLP and self-attention as associative memory stores. It estimates how many key-value pairs each can hold before interference drowns the signal.