TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Alex Wang’s article explains transformer components through associative memory: layers store key-value associations and retrieve values from similar inputs. It estimates that, with 64-dimensional keys, a basic linear memory can store about 64 memories before interference matches the target signal, an MLP about 2,700, and self-attention more than one million under the post’s assumptions. These figures are explanatory estimates, not universal capacity guarantees.
Alex Wang has published an explainer arguing that transformer layers can be understood as memory stores: they encode key-value associations and retrieve values when a query resembles a stored key. Using an estimate based on 64-dimensional keys, Wang compares a basic linear store at roughly 64 memories with an MLP at about 2,700 and self-attention at more than one million, illustrating how different mechanisms can reduce recall interference.
Wang’s central measure is the interference-to-signal ratio: how much unrelated stored information affects the recall of a target memory. A store’s capacity, in this framework, is the number of associations it can hold before interference becomes as large as the signal from the requested memory. The piece treats a memory as a key-value pair: the key supports lookup, while the value is the information returned.
In the basic linear example, each association is added to a weight matrix as an outer product of its value and key. A query is compared with all stored keys, and the returned output is the sum of their values, weighted by similarity. Wang reasons that, for random unit-length keys and similarly scaled random values, interference grows roughly with the square root of the number of memories divided by the key dimension. That gives an estimated capacity of about the key dimension—roughly 64 memories for 64-dimensional keys.
The article then describes ways to improve on that basic store. Its stated comparison is about 2,700 memories for an MLP and more than a million for self-attention, again with 64-dimensional keys. The post presents these as heuristics drawn from its memory model, not as a claim that every trained model reliably recalls that many arbitrary facts.
Why Memory Capacity Matters
The framing offers a way to reason about why neural networks can retain learned information yet sometimes return blended or inaccurate answers. The issue is not simply whether information was written into parameters or a context: it is also whether the retrieval mechanism can distinguish the requested association from competing ones. Wang’s interference model makes that trade-off easier to discuss across different layer types.
The comparison also gives readers a conceptual bridge between model weights and attention. A linear layer stores associations in fixed parameters, while self-attention can match a query against a collection of keys directly. That distinction helps explain why attention can retrieve from many entries in a sequence without requiring every key to occupy a distinct orthogonal direction in a fixed-dimensional space. The practical relevance is to interpreting model design and memory claims, not to measuring the factual knowledge of a particular commercial chatbot.
For researchers and practitioners, the figures may serve as a starting point for asking how architecture, key dimension and similarity affect retrieval. They should not be read as a direct forecast of how many facts a deployed model can answer correctly: real training data, learned representations, repeated or correlated keys, value structure and evaluation methods all matter.
As an affiliate, we earn on qualifying purchases.
From Linear Layers to Attention
Wang begins with an everyday example of asking a language model to recall the rules of a board game, then turns to the broader question of how models retrieve information. The source describes an associative memory as a function that stores key-value pairs and returns a value when given a matching or similar query. It focuses on recall quality rather than merely the amount of information that can be written.
For the linear store, the article assumes normalized keys and considers random, similarly scaled keys and values. If keys were orthogonal, a query could retrieve its matching value without interference. But a key space with dimension 64 has only 64 mutually orthogonal directions; beyond that, similarities among keys contribute unwanted values to a recall. Wang’s estimate follows from how these contributions accumulate under the stated random-vector assumptions.
The post also connects memory-writing to learning: a gradient-descent update to a linear layer changes its weights by an outer product of an input and an output gradient. Wang notes that this can be interpreted as writing an association into the layer. The source then moves from this simple mechanism toward MLPs and self-attention, presenting each as a way to improve storage or retrieval rather than treating all neural-network layers as interchangeable.
“The key property that characterizes a “good” memory store is its ability to recall what it stored.”
— Alex Wang
As an affiliate, we earn on qualifying purchases.
Limits of the Capacity Estimates
The supplied source gives a theoretical, intuitive comparison, but does not establish that the stated capacities apply to every model or task. Its initial estimate relies on simplified assumptions about random keys and values, normalized key lengths and how interference accumulates. The excerpt does not provide benchmark results showing that trained networks reach these capacities on real-world recall tests.
The source material ends as Wang begins discussing methods to reduce interference, so the full derivation and qualifications behind the later MLP and attention figures are not available here. It is also unclear from the supplied material what conditions define a “memory” for those estimates, how sharply recall quality degrades near each threshold, or how results change with learned, correlated representations. The figures should be attributed to Wang’s analysis and not presented as universal model limits.
As an affiliate, we earn on qualifying purchases.
Further Details in the Full Analysis
The next step for readers is to consult Wang’s full post for the complete derivations and any additional assumptions behind the MLP and self-attention comparisons. The supplied excerpt does not announce a follow-up experiment, publication date or external validation, so no such development is confirmed. A practical test of the framework would compare recall performance across layer types while varying key dimension, number of stored associations and similarity among keys.
Until such evidence is available, the central takeaway is conceptual: storage and reliable recall are different problems, and interference can limit a memory system even when its parameters can hold many associations. Wang’s numerical examples give readers a way to compare mechanisms, but they remain estimates from the article rather than measured guarantees about current language models.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the actual development?
Alex Wang published an explainer that models linear layers, MLPs and self-attention as associative memory stores and compares their estimated recall capacities.
What does the article mean by memory capacity?
It is the number of key-value associations a store can contain before interference from other memories becomes as large as the signal from the memory being queried. It is a measure within Wang’s framework, not a general count of facts a model knows.
What capacity figures does Wang give for 64-dimensional keys?
The post gives estimates of about 64 memories for a basic linear store, about 2,700 for an MLP and more than one million for self-attention. They depend on the article’s assumptions and should not be treated as universal guarantees.
Does this show how many facts a chatbot can recall?
No. The figures describe a simplified memory model. The supplied source does not establish that a deployed chatbot can reliably recall those numbers of arbitrary facts; training, representation, data and testing conditions can affect real-world performance.
Source: hn
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
