Attention moves information between token positions. The MLP transforms features within each position. Residual connections preserve the existing state while both sublayers contribute updates that later layers can read.
Blue bars are the residual vector. Cyan bars are the attention update, pink bars are the MLP update, and yellow gates mark normalization.
A residual connection adds a sublayer output to the state that entered it. The sublayer can contribute a correction without reconstructing the full representation. Later attention heads and MLPs read the accumulated stream and add their own updates.
The identity path carries information and gradients around each nonlinear transformation. A sublayer can learn a small useful update while the original signal remains available.
Queries select weighted combinations of value vectors from other positions. The resulting update can copy, route, or combine contextual information.
The feed-forward network expands, activates, and projects the current position. It can detect combinations of features and write a transformed result back.
LayerNorm or RMSNorm standardizes the input seen by a sublayer. Placement changes optimization behavior and the scale trajectory of the residual stream.
A vector per token position that persists through the network. It is not a single human-readable concept; many features can be represented across directions and magnitudes.
Persistent stateBuilds each token's update from other positions. Multiple heads can implement different routing patterns, but their outputs are combined before the residual addition.
Across positionsApplies the same learned nonlinear transformation independently at each position. Expansion provides a larger intermediate feature space before projection back to model width.
Within positionPre-norm feeds normalized state into each sublayer while preserving an unnormalized identity stream. Post-norm normalizes after residual addition. Their training dynamics differ.
Scale controlMany architecture diagrams hide the residual addition inside arrows. Stepping through the state makes clear which operation changes the stream and which merely prepares an input.
In a pre-norm block, normalization creates the attention input without replacing the residual stream itself.
attention_input = norm(residual)
Attention reads all relevant token positions and produces one update vector for this position. The update is not yet the new token state.
attention_write = attention(attention_input)
The attention write joins the stream. A second normalized read enters the MLP, whose nonlinear feature update is added in turn.
residual += attention_write residual += mlp(norm(residual))
The next block receives the accumulated vector. With enough layers, update scale and normalization determine whether features remain stable, drift, or dominate.
It is temporary activation state for the current sequence, not durable memory. KV cache stores past attention keys and values; external agent memory lives outside the model.
Head outputs are concatenated or combined, projected through an output matrix, and then the resulting vector is added to the residual stream.
A small update in a behaviorally important direction can matter even when the overall vector direction changes little. Global geometric summaries cannot identify every feature.
They place normalization on different paths. Pre-norm often improves gradient flow in deep transformers, while post-norm changes the scale and representation after each addition.
Explore SuperPowers workflows that turn explicit goals into visible agent actions, from tool use and computer state to complete website generation.
Explore SuperPowersWebsite-building agentComputer-use cache