- This article explains how large language models store and retrieve vast amounts of information. The author views the model's layers as a memory device that pairs cues (keys) with the contents to retrieve (values), and calls the structure that retrieves contents using similar cues associative memory.
- The simplest linear layer builds memory by multiplying keys by values and adding them up. The author estimates that, assuming 64-dimensional keys, this method can reliably retrieve only about 64 memories.
- The author holds that MLPs raise storage capacity and recall accuracy together, and explains that self-attention recalls very accurately and has no clear theoretical upper limit on the number of stored items. Under the same 64-dimensional key condition, the figures of about 2,700 and 1 million or more are also estimates the author presents.
- As more memories accumulate, the degree to which other memories interfere with recall grows. The article uses this interference-to-signal ratio as the key metric for comparing the capacity of each structure.
O: This suggests a scene where a graduate student who wants to work out for themselves how associative memory operates calculates by hand the 64-item limit of a linear layer.
X: The capacity figures 64, 2,700, and 1 million or more are the author's theoretical estimates that assume 64-dimensional keys, so they are hard to treat as measurements from an actual model.