Transformers: Attention, Architecture, Training, and Scaling

Before we talk about attention, it is worth naming the problem it was designed to solve. A sequence model is not merely a machine that consumes tokens in order; it is a machine that must route information between positions. If token $x_i$ matters for predicting something at position $j$, the architecture needs a reliable computational path from $i$ to $j$. The central question is: how long, fragile, and sequential is that path?

Many important tasks can be written abstractly as sequence-to-sequence mappings,

$$x_{1:n} \mapsto y_{1:m},$$

where the input and output lengths may differ. Translation maps a sentence in one language to a sentence in another. Summarization maps a long document to a shorter text. Code generation maps a prompt or partial program to a completed program. Even when the output is not explicitly a separate sequence, language modeling has the same flavor: at each position $t$, the model predicts the next token from the previous context,

$$p_\theta(x_t \mid x_{<t}).$$

This notation hides the hard part. The conditioning set $x_{<t}$ may be large, but not every previous token is equally relevant. A model predicting the verb in a sentence may need to find the true subject many tokens earlier. A model completing code may need to remember an opening bracket, variable declaration, or function signature hundreds or thousands of tokens back. A translation model may need to align a word near the end of the source sentence with a word near the beginning of the target sentence. In all cases, sequence modeling requires selective communication between positions.

Classical recurrent neural networks handle this by passing information through a hidden state:

$$h_t = f_\theta(h_{t-1}, x_t).$$

This is elegant because it respects temporal order and can in principle summarize everything seen so far. But it creates a narrow communication channel. If information from $x_i$ is needed at $x_j$, it must survive repeated transformations through

$$h_i \rightarrow h_{i+1} \rightarrow \cdots \rightarrow h_j.$$

The number of computational steps between the two positions grows with their distance, roughly $L = |j-i|$. Gradients must also travel through this same chain during training. Gating mechanisms such as LSTMs and GRUs reduce the damage, but they do not remove the fundamental bottleneck: distant tokens communicate through a long sequential path.

Convolutional sequence models improve parallelism because all positions in a layer can be processed simultaneously. However, local convolutions have their own routing problem. A kernel of small width only mixes nearby tokens in one layer, so long-range interaction requires stacking many layers. Dilated convolutions shorten the path, but the architecture still imposes a predefined communication pattern. Whether two positions can exchange information efficiently depends on the convolutional design rather than on the content of the sequence itself.

This suggests three desiderata for a strong sequence architecture:

• Relevant conditioning: each output should be able to depend on the input or context positions that matter.
• Parallel training: computations across positions should not be forced into a strict left-to-right loop when the training targets are already known.
• Short path length: the number of computational steps $L$ between any two positions should be small, ideally constant or close to constant.

The key insight behind Transformers is to replace sequential recurrence with learned content-based communication. Instead of requiring information to move one step at a time through a hidden state chain, each position can directly ask: which other positions contain information useful for me? Attention implements this as a differentiable retrieval mechanism. Positions produce queries, keys, and values; similarity between queries and keys determines where information flows. The route is not hard-coded by distance or adjacency. It is learned from content.

This matters because many sequence dependencies are sparse but not local. A token may need its immediate neighbors for syntax, a faraway noun for agreement, and an even farther definition for semantic interpretation. Architectures based only on local or sequential propagation must repeatedly carry all potentially useful information forward. Attention instead allows the model to create direct edges between relevant positions, making the effective path length between $i$ and $j$ very short.

The visual below condenses this bottleneck into a single picture: an input sequence $x_{1:n}$, an output sequence $y_{1:m}$, and the central challenge of connecting distant but relevant positions. The faint recurrent chain represents the older strategy: information moves through many local transitions, producing a long path $L$. The highlighted long-range arrow represents the dependency we actually care about.

The same visual also previews the Transformer solution. Rather than relying only on neighboring steps, positions exchange information through learned, content-based links. Those direct communication paths are the conceptual bridge from traditional sequence models to attention: the model still respects sequence structure, but it no longer forces all information to travel through a narrow sequential corridor.

The bottleneck is easiest to dismiss when we talk about it abstractly: “long-range dependency” sounds like a rare linguistic edge case. But the problem appears in one of the most ordinary tasks a language model performs: choosing the next word. Even a short sentence prefix can force the model to decide whether to trust nearby evidence or route information from a more distant but grammatically relevant token.

Consider the prefix

$$x_{<t}=\text{``The keys to the cabinet near the door''}.$$

The next word should be “are”, not “is”:

$$\text{``The keys to the cabinet near the door are ...''}$$

The grammatical subject is keys, which is plural. But by the time the model reaches the prediction position, the most recent nouns are cabinet and door, both singular. A model that overweights local context may be tempted by the nearby phrase “near the door” and predict a singular verb:

$$\text{``The keys to the cabinet near the door is ...''}$$

This is the long-range agreement trap. The correct prediction depends not on the closest noun, but on the noun that structurally controls the verb. In the prefix,

$$x_2=\text{``keys''}$$

is the controller, while

$$x_5=\text{``cabinet''}, \qquad x_8=\text{``door''}$$

are distractors. The model must learn a conditional preference of the form

$$p_\theta(x_t=\text{``are''}\mid x_{<t}) > p_\theta(x_t=\text{``is''}\mid x_{<t}).$$

The important point is that this inequality is not merely about memorizing that “keys” often goes with “are.” It requires the model to identify which earlier token is relevant for this prediction. The phrase contains multiple nouns, and the nearest ones are misleading. Sequential distance and grammatical relevance have come apart.

This exposes a weakness of purely local prediction. If the model primarily summarizes recent tokens, then the final phrase “near the door” dominates the representation near the prediction point. But the word door should not control the verb. It is embedded inside a prepositional phrase modifying cabinet, which itself is embedded inside another prepositional phrase modifying keys. The relevant dependency skips over these intervening tokens.

A good sequence model therefore needs a mechanism for content-based routing. Instead of asking, “Which token is closest to the current position?”, it should ask something more like:

• Which previous token is syntactically or semantically relevant?
• Which token supplies the feature needed for this prediction?
• Which apparent cues are merely distractors because of their local proximity?

For subject–verb agreement, the feature being routed is number: plural versus singular. In other examples, it might be entity identity, coreference, topic, tense, quotation state, or a variable binding. The general pattern is the same: the model must retrieve information by relevance, not by position alone.

This is one of the motivations for attention. Attention will eventually give us a differentiable way to compare the current prediction context against earlier token representations and assign high weight to the tokens whose content matters. The long-range token does not need to be compressed through every intermediate step with equal fidelity; it can be selected directly when it becomes useful.

The visual below condenses this failure case into a single next-token decision. The prefix is laid out as token boxes, with the blank prediction position at the end. The plural noun keys is far away but is the true controller of the verb, while cabinet and door are closer distractors. The central mistake to avoid is treating proximity as a proxy for relevance.

The probability inequality on the right states the learning target: the model should assign higher probability to “are” than to “is” given the whole prefix. That small inequality captures the larger architectural lesson: long-range dependencies require mechanisms that can route information by content relevance, not merely by sequential neighborhood.

The agreement trap from the previous section is not just a quirky linguistic example; it exposes a more general systems problem. A model must move information from some earlier position $i$—where the relevant subject, entity, or condition appears—to a later position $j$, where that information is needed to make a prediction. The intervening tokens may be syntactically plausible distractors, but semantically irrelevant. The question is: how many computational steps must the signal traverse before position $j$ can use what position $i$ knew?

For a recurrent model, the route is built into the architecture. Information at $x_i$ is first absorbed into a hidden state $H_i$, then passed forward one position at a time:

$$x_i \rightarrow H_i \rightarrow H_{i+1}\rightarrow \cdots \rightarrow H_j, \qquad L=\mathcal{O}(|j-i|).$$

This gives RNNs a useful inductive bias: nearby temporal continuity is natural, and the hidden state acts like a running summary. But the same design becomes a bottleneck for long-range dependencies. If $i$ and $j$ are far apart, then the representation of $x_i$ must survive many state updates, each of which may overwrite, compress, or distort it. Even if the model has gates, such as in an LSTM or GRU, the route is still sequential. The architecture can learn to preserve information, but it cannot avoid the fact that the information must pass through many intermediate states.

The training signal suffers from the same geometry. When a loss at position $j$ assigns credit to something that happened near $i$, the relevant derivative contains a long product of Jacobians:

$$\frac{\partial H_j}{\partial H_i} = \prod_{t=i+1}^{j} \frac{\partial H_t}{\partial H_{t-1}}.$$

This product is the mathematical heart of the vanishing and exploding gradient problem. If the typical singular values of these Jacobians are smaller than one, the gradient decays exponentially with distance; if they are larger than one, it can blow up. Gating, normalization, careful initialization, and gradient clipping can make this more manageable, but they do not remove the long credit-assignment path. The model is still being asked to propagate both information and gradients through $\mathcal{O}(|j-i|)$ sequential transformations.

Convolutional sequence models attack the problem differently. Instead of carrying a hidden state from left to right, they update all positions in parallel using local windows. This is excellent for parallel training: every position in a layer can be computed at the same time. But locality introduces a different limitation. A token can only influence positions within the receptive field of the convolution, and that receptive field grows layer by layer. With a kernel of fixed width, distant positions require many layers before they can interact.

So CNNs trade the RNN’s sequential time bottleneck for a depth bottleneck. A sufficiently deep convolutional network can connect distant positions, and dilated convolutions can expand the receptive field faster, but the architecture still imposes a structured route through intermediate neighborhoods. Distant communication is possible only after repeated mixing. In practice, this means either many layers, large kernels, carefully chosen dilation schedules, or some combination of these. The model’s ability to relate $x_i$ and $x_j$ is mediated by architectural distance.

This is the motivation for attention as a new primitive. Instead of forcing information to travel through every intermediate hidden state, or to diffuse through local convolutional neighborhoods, we would like position $j$ to directly retrieve information from position $i$ when $i$ is relevant. In the idealized path-length sense, attention offers:

$$L = \mathcal{O}(1).$$

That statement does not mean attention is computationally free. Full self-attention over a sequence of length $n$ compares many pairs of positions, which introduces its own cost. Rather, the point is about communication distance: once the attention scores are computed, any position can incorporate information from any other position in a single layer. The burden shifts from “can the information survive the route?” to “can the model learn which positions matter?”

This shift is subtle but crucial. RNNs and CNNs bake in a strong notion of locality: information moves through adjacent time steps or nearby windows. Attention weakens that assumption and replaces it with content-based routing. If a verb at position $j$ needs the subject at position $i$, the model can learn to assign high weight to that subject directly, even across many distractors. The resulting architecture is often easier to optimize for long-range dependencies because the forward information path and the backward credit-assignment path are both shorter.

The comparison can be summarized as follows:

• RNNs have long sequential paths and long gradient products.
• CNNs parallelize well, but distant interactions require depth or dilation.
• Attention permits direct pairwise communication, making long-range dependency modeling a question of learned retrieval rather than repeated propagation.

The visual below condenses this argument into a side-by-side comparison. The recurrent row emphasizes the chain from $x_i$ to $H_j$, while the gradient row highlights why that chain is also an optimization problem. The convolutional row captures receptive-field growth through stacked local windows. The attention row contrasts these with a direct connection from $i$ to $j$, foregrounding the key design goal: reduce the path length for relevant interactions to $\mathcal{O}(1)$.

Read the table not as saying that attention is universally cheaper or always better, but as isolating the architectural reason Transformers became compelling. They replace forced sequential or local communication with a differentiable mechanism for deciding who should talk to whom. That mechanism—attention as learned lookup—is the next object we will derive.

The previous discussion identified a common bottleneck behind both recurrence and convolution: routing information. If a token near the end of a sequence needs evidence from a token near the beginning, an RNN must carry that evidence through many recurrent steps, while a CNN must propagate it through many local receptive fields unless the network is made very deep or uses large kernels. The issue is not merely that long paths are inconvenient; long paths make learning fragile. Gradients, intermediate states, and local transformations all become part of the communication channel.

A more direct primitive would let each position ask: which other positions contain information useful for me right now? Instead of forcing information to move step by step through a fixed computational graph, we want every position to be able to retrieve relevant content from anywhere in the sequence in one operation. This is the motivation behind attention.

The simplest abstraction is a content-based lookup. Imagine that each position $j$ in a sequence stores some information in a vector $v_j$, called a value vector. Now suppose position $i$ wants to update its representation. Rather than reading only its neighbor or the previous hidden state, position $i$ forms a weighted combination of all available values:

$$\text{retrieved information for position } i = \sum_{j=1}^{n} a_{ij}v_j.$$

The coefficients $a_{ij}$ are the attention weights. They say how much position $i$ retrieves from position $j$. To make this retrieval behave like a soft selection, the weights are constrained to be nonnegative and sum to one:

$$a_{ij}\ge 0, \qquad \sum_{j=1}^{n} a_{ij}=1.$$

These constraints make the retrieved vector a convex combination of the value vectors. Intuitively, position $i$ is not copying a single source exactly; it is averaging information from multiple sources, with more relevant positions receiving larger weights. This convex-mixture view is important because it gives attention a stable numerical interpretation: the output remains in the span of available information rather than becoming an unconstrained linear explosion.

A hard lookup would choose one position $j$ and return $v_j$. That is useful in classical data structures, but it is awkward for gradient-based learning: a discrete choice is not smoothly differentiable with respect to the scores that produced it. Attention replaces that hard decision with a soft lookup. If the model is uncertain between two relevant tokens, it can assign weight to both. If training later reveals that one source was more useful, gradients can continuously shift probability mass toward it.

This also explains why attention is more than a memory trick. The weights should not be fixed by distance or position alone. They should depend on content compatibility: what the current position needs and what each candidate source offers. For example, in a translation model, a decoder position producing a verb may need to retrieve the subject from far away. In a language model, a pronoun may need to retrieve a compatible antecedent. The useful source is determined by meaning and context, not simply by being nearby.

There are a few subtle assumptions hidden in this primitive. First, the value vectors $v_j$ must contain information worth retrieving; attention can route information, but it cannot recover content that was never represented. Second, the weights $a_{ij}$ must be produced by a learnable scoring mechanism that can compare position $i$'s needs with position $j$'s contents. Third, because the retrieved vector is a mixture, attention can sometimes blur information when many incompatible sources receive nontrivial weight. Later, scaled dot-product attention will address how to compute these weights effectively and how to keep the scoring distribution numerically well behaved.

The key conceptual shift is therefore:

• RNN/CNN view: information moves through a sequence by fixed local computation paths.
• Attention view: each position directly retrieves a weighted mixture from all positions.
• Learning problem: choose the weights $a_{ij}$ from content, so useful tokens communicate strongly even when distant.

The visual below condenses this idea into a single routing picture. The sequence positions each hold a value vector $v_j$. For a target position $i$, arrows from all positions represent possible communication channels, and their thickness represents the learned attention weights $a_{ij}$. The output at position $i$ is not one copied vector, but the soft mixture $\sum_j a_{ij}v_j$.

This compact diagram is the bridge from motivation to mechanism. We have not yet specified how the weights are computed—that will require embeddings, queries, keys, values, and scaling—but we have specified the primitive we want: differentiable, content-based retrieval over a sequence.

Now that we have framed attention as a kind of differentiable lookup, we need to answer a deceptively basic question: what exactly are we looking up with? A Transformer cannot operate directly on raw symbols like "cat", "sat", or ".". Those symbols are discrete vocabulary items; they have no geometry, no dot products, no notion of similarity that a neural network can manipulate smoothly. Before attention can compare one token to another, every token must be represented as a vector in a shared continuous space.

The first ingredient is the token embedding table. If the vocabulary is $\mathcal{V}$ and the model width is $d_{\mathrm{model}}$, we learn a matrix

$$E\in\mathbb{R}^{|\mathcal{V}|\times d_{\mathrm{model}}}.$$

Each row of $E$ is the learned vector representation of one vocabulary item. For a token $x_i$, the embedding lookup returns

$$e(x_i)\in\mathbb{R}^{d_{\mathrm{model}}}.$$

This is often described as a “lookup,” but it is still part of the differentiable model: the selected embedding vector participates in the forward computation, receives gradients during backpropagation, and is updated during training. Over time, the model learns an embedding geometry in which useful distinctions for prediction become linearly accessible to later layers.

However, token identity alone is not enough. The sequence

$$\text{“dog bites man”}$$

does not mean the same thing as

$$\text{“man bites dog”}.$$

If we gave self-attention only the multiset of token embeddings $\{e(x_1),\ldots,e(x_n)\}$, then the model would know which tokens appeared, but not where they appeared. This is especially important because vanilla self-attention, unlike recurrence or convolution, does not inherently process tokens left-to-right or through local neighborhoods. Its core operation compares rows of a matrix to other rows of the same matrix. Without additional positional information, that operation is naturally permutation-equivariant: reorder the input rows, and the corresponding outputs reorder in the same way.

So Transformers add a second vector to each token representation: a position vector. For a maximum sequence length $n$, we can represent these position vectors as rows of a matrix

$$P\in\mathbb{R}^{n\times d_{\mathrm{model}}}, \qquad p_i\in\mathbb{R}^{d_{\mathrm{model}}}.$$

The vector $p_i$ tells the model that this row corresponds to position $i$. In the original Transformer, these were sinusoidal encodings; in many modern models, they are learned or replaced with relative/rotary variants. But the conceptual role is the same: position information must enter the representation somehow, because attention by itself only sees a set of content vectors.

The simplest absolute-position construction is additive. For each token $x_i$, we combine “what token is here” with “where it is”:

$$x_i \quad\longmapsto\quad e(x_i)+p_i.$$

Stacking these row vectors gives the Transformer’s input matrix

$$X= \begin{bmatrix} e(x_1)+p_1\\ e(x_2)+p_2\\ \vdots\\ e(x_n)+p_n \end{bmatrix} \in\mathbb{R}^{n\times d_{\mathrm{model}}}.$$

This matrix $X$ is the object that will be projected into queries, keys, and values in the next step. Each row is one token position, and each row lives in the same $d_{\mathrm{model}}$-dimensional space.

The addition $e(x_i)+p_i$ is worth pausing over. We are not concatenating token and position into a larger vector; we are superimposing them in the same model dimension. That means the model must learn to use the shared coordinates to encode both lexical/content information and positional information. This works because the subsequent linear projections can learn directions that respond to content, position, or mixtures of the two. But it also encodes an assumption: the model width $d_{\mathrm{model}}$ must be large enough to carry all the information the network needs.

A useful way to think about a row of $X$ is:

• Embedding component: “what symbol or subword is present?”
• Position component: “where in the sequence does it occur?”
• Combined representation: “what is at this location?”

That last phrase is crucial. Attention will not retrieve information from isolated word types; it will retrieve from contextualizable slots in a sequence. The same token can appear in two positions and begin with the same embedding, but after adding different $p_i$'s, the initial vectors are no longer identical. This allows later attention layers to distinguish, for example, the first occurrence of a word from the second.

There is also an important failure mode hiding here. If we omitted $p_i$, then a self-attention layer with shared projections would not know whether a token came first, last, or somewhere in the middle. It could still compare content, but it would lack sequence order. For tasks where order matters—and almost all language tasks do—that is a severe limitation. Positional information is therefore not a decorative add-on; it is what turns a bag of token vectors into a sequence representation.

The visual below compactly summarizes this construction as a pipeline: raw tokens are mapped through the embedding table $E$, position rows are supplied from $P$, and corresponding vectors are added row by row to form $X\in\mathbb{R}^{n\times d_{\mathrm{model}}}$. The key idea to look for is that the Transformer input is not “just embeddings,” but embeddings plus positions.

It also foreshadows the next step. Once $X$ has been assembled, attention can create queries, keys, and values by learned linear projections. In other words, the differentiable retrieval mechanism we want is built on top of this matrix: each row of $X$ is now a content-and-position-aware record that attention can compare, weight, and combine.

Once tokens have been turned into vectors, we have a useful representation at every sequence position—but each position is still mostly carrying local information: “what token am I?” and “where am I?” The next problem is how one position can use information stored at other positions. If the word “it” appears late in a sentence, its representation may need to borrow meaning from a noun many tokens earlier. If a code variable is used after several lines, its current representation should be able to retrieve the earlier definition.

The key idea of attention is to make this retrieval differentiable. Instead of choosing exactly one previous token or one memory slot, a position forms a weighted average over many stored vectors. For a particular query position $i$, suppose every position $j \in \{1,\dots,n\}$ stores some vector $v_j$, called a value. The output representation at position $i$ is

$$z_i = \sum_{j=1}^{n} a_{ij} v_j .$$

This is the central retrieval equation. The output $z_i$ is not copied from a single location; it is blended from all available values. The coefficient $a_{ij}$ tells us how much position $i$ retrieves from position $j$.

For this weighted average to behave like retrieval, the weights should be nonnegative and should sum to one:

$$a_{ij} \ge 0, \qquad \sum_{j=1}^{n} a_{ij}=1.$$

So $z_i$ lies in the convex hull of the value vectors. Intuitively, attention says: “construct the new representation by mixing stored content, with mixture proportions determined by relevance.” This is a softer and more trainable version of lookup in a dictionary or memory table.

The weights themselves come from compatibility scores $s_{ij}$, where $s_{ij}$ measures how relevant position $j$ is to position $i$. We convert these arbitrary real-valued scores into normalized weights using a softmax over the retrievable positions:

$$a_{ij} = \frac{\exp(s_{ij})}{\sum_{u=1}^{n}\exp(s_{iu})}.$$

The denominator is important: for a fixed retrieving position $i$, the model compares all candidate positions $j$ against one another. Attention is therefore relative: a token receives high weight not merely because its score is large in isolation, but because its score is large compared with the alternatives in the same row.

This row-wise normalization has several consequences. First, attention weights are easy to interpret as a distribution over source positions. Second, the operation is differentiable end-to-end, so learning can adjust both the scoring mechanism and the stored value vectors. Third, the softmax introduces competition: increasing $a_{ij}$ tends to decrease the mass assigned elsewhere. That competition is part of what makes attention behave like selective retrieval rather than an unstructured sum.

There is also a subtle but crucial separation here: scoring and content are conceptually different. The score $s_{ij}$ determines where to look; the value $v_j$ determines what information is retrieved once we look there. This distinction will become central when we introduce queries, keys, and values. For now, it is enough to notice that the vector used to decide relevance need not be identical to the vector whose information is ultimately copied into the output.

Stacking the outputs for all positions gives the matrix form. Let $V \in \mathbb{R}^{n \times d_v}$ contain the value vectors row by row, and let $A \in \mathbb{R}^{n \times n}$ contain the attention weights, with row $i$ holding the distribution $(a_{i1},\dots,a_{in})$. Then

$$Z = A V.$$

This equation is compact but powerful. Each row of $Z$ is a weighted average of the rows of $V$. The matrix $A$ is not just a generic linear map; it is row-stochastic, meaning each row is a probability distribution. Attention is therefore a structured, data-dependent linear combination of stored representations.

The visual below condenses this idea into two complementary views. On the algebraic side, it emphasizes the chain from scores $s_{ij}$, to softmax-normalized weights $a_{ij}$, to retrieved output $z_i$. On the retrieval side, one position $i$ sends different-strength connections to several stored values $v_j$, and those weighted contributions merge into the output vector.

The most important takeaway is that attention is not yet “magic Transformer machinery.” At this stage, it is simply weighted differentiable retrieval. The model computes relevance scores, normalizes them into a distribution, and uses that distribution to average content vectors. The next step is to specify how the scores $s_{ij}$ are produced—and that is where queries, keys, and values enter.

The weighted-retrieval view gives us a useful abstraction: once we have an attention matrix $A$, producing outputs is just

$$Z = AV,$$

a weighted average of “retrievable” vectors. But this leaves an important question unresolved: where do the weights in $A$ come from, and what exactly are we averaging? If the same representation of a token is used both to decide whether it is relevant and to supply the content returned, the model is forced to entangle two different roles. Transformers avoid this by separating matching from retrieval.

The key move is to project each input position into three learned spaces:

$$Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.$$

Here $X\in\mathbb{R}^{n\times d_{\mathrm{model}}}$ is the sequence of contextual token representations: $n$ positions, each represented by a $d_{\mathrm{model}}$-dimensional vector. The learned projection matrices have shapes

$$W_Q,W_K\in\mathbb{R}^{d_{\mathrm{model}}\times d_k}, \qquad W_V\in\mathbb{R}^{d_{\mathrm{model}}\times d_v},$$

so the resulting matrices are

$$Q,K\in\mathbb{R}^{n\times d_k}, \qquad V\in\mathbb{R}^{n\times d_v}.$$

Conceptually, each row of these matrices plays a different role. The row $q_i$ of $Q$ is the query for position $i$: it represents what that position is looking for. The row $k_j$ of $K$ is the key for position $j$: it represents how position $j$ advertises itself for matching. The row $v_j$ of $V$ is the value for position $j$: it is the information returned if some other position decides to attend to $j$.

This distinction is subtle but central. A word may need to be matched according to one set of features, while returning a different set of features once selected. For example, a pronoun might look for a compatible noun phrase using syntactic and semantic cues encoded in queries and keys, but the information retrieved from the noun phrase might include number, gender, entity identity, or broader contextual meaning encoded in the value vector. The model should not have to use the same coordinates for all of these purposes.

In self-attention, all three matrices are projected from the same input $X$. That is why the mechanism is “self”: every position can attend to other positions in the same sequence. But the projections are different, so “same source” does not mean “same representation.” The model learns three views of each token:

• Query view: what this position needs.
• Key view: how this position can be found.
• Value view: what this position contributes if selected.

This separation also explains why attention is more flexible than a fixed similarity computation on raw embeddings. If we compared rows of $X$ directly, the notion of relevance would be tied to whatever features happen to be present in the model representation. By learning $W_Q$ and $W_K$, the Transformer learns a task-specific compatibility space. By learning $W_V$, it separately learns what content should flow forward after compatibility has been determined.

There are also important dimensional choices hidden in these equations. The query and key dimensions must match, because they will be compared to produce attention scores; hence both live in $\mathbb{R}^{d_k}$. The value dimension $d_v$, however, need not equal $d_k$, because values are not used for matching in the same way. They are aggregated after the attention weights have already been computed. In practice, architectures often choose convenient equal dimensions, especially inside multi-head attention, but the mathematical roles remain distinct.

A useful way to think about the full computation is:

$$\text{queries and keys determine } A, \qquad \text{values determine what } A \text{ averages.}$$

So $Q$ and $K$ answer the question, “Which positions should interact, and how strongly?” while $V$ answers, “What information should be passed along once that interaction is chosen?” This is the bridge from abstract weighted retrieval to the concrete attention mechanism used in Transformers.

The visual below compactly summarizes this decomposition. A single input matrix $X$ branches into three learned projections: $Q$, $K$, and $V$. The parallel arrows emphasize that these are not three different input sequences, but three learned views of the same sequence. Highlighting individual rows $q_i$, $k_j$, and $v_j$ reinforces the position-wise interpretation: one position asks, another position matches, and the matched position returns content.

The faded continuation toward $Z=AV$ is also important. It reminds us that values are still the objects being averaged, as in weighted retrieval, but the weights $A$ will now be produced by comparing queries and keys. This separation is the conceptual step that makes scaled dot-product attention possible: first learn what to match, then learn what to return.

With queries, keys, and values in place, the next question is: how should a query decide which keys are relevant? We have already separated “what I am looking for” ($q_i$) from “what each position offers as an address” ($k_j$) and “what content I can retrieve” ($v_j$). Attention now needs a compatibility function that turns each query–key pair into a scalar score.

The simplest and most important choice is the dot product:

$$s_{ij}=q_i^\top k_j.$$

This score is large when $q_i$ and $k_j$ point in compatible directions. Geometrically, the dot product rewards alignment: if two vectors have similar directions, their inner product is positive and large; if they are orthogonal, it is near zero; if they point in opposing directions, it can be negative. In attention, this means position $i$ assigns a high raw score to position $j$ when the learned query at $i$ matches the learned key at $j$.

There is a subtle but important assumption here: the learned projections that produce $q_i$ and $k_j$ are free to shape the space in which “matching” happens. We are not comparing raw token embeddings directly. Instead, the model learns a coordinate system where certain directions correspond to useful retrieval patterns: syntactic agreement, coreference, local continuation, delimiter matching, or any other relation that helps the task. The dot product is simple, but the learned projections make it expressive.

For a sequence of length $n$, every query position compares itself to every key position. If we stack the query vectors into a matrix

$$Q \in \mathbb{R}^{n \times d_k}$$

and the key vectors into

$$K \in \mathbb{R}^{n \times d_k},$$

then all $n^2$ pairwise dot products appear at once in the matrix product

$$[\,s_{ij}\,]_{ij}=QK^\top, \qquad (QK^\top)_{ij}=q_i^\top k_j.$$

This is one of the central computational facts behind Transformers: attention is vectorized all-pairs comparison. Instead of iterating through the sequence recurrently, each position can compare against every other position in a single batched matrix multiplication. That is why attention is so well matched to modern accelerators: the communication pattern is dense, but it is expressed as large linear algebra operations.

The raw score matrix $QK^\top$ is not yet a retrieval distribution. Each row contains the scores for one query position $i$ against all possible source positions $j$. To turn these scores into weights, we apply a row-wise softmax:

$$A=\operatorname{softmax}(QK^\top).$$

Here $A_{ij}$ can be read as “how much position $i$ attends to position $j$.” The rows of $A$ sum to one, so each query forms a weighted average over values. The retrieved output vectors are then

$$Z=AV = \operatorname{softmax}(QK^\top)V.$$

This is the full dot-product attention pattern before scaling and masking: compare queries to keys, normalize scores into attention weights, then use those weights to average values.

It is worth emphasizing what this buys us. A recurrent model must pass information through a chain of hidden states, so distant positions interact through many sequential steps. A convolutional model needs either large kernels or many layers to connect distant tokens. Dot-product attention, by contrast, gives every position a direct path to every other position in one layer. The cost is that the score matrix has $n^2$ entries, so dense self-attention is powerful but expensive for long sequences.

There are also important failure modes hidden in this compact formula:

• If dot products become too large in magnitude, the softmax can become extremely peaked, causing gradients through most positions to shrink.
• If all dot products are similar, the attention weights become nearly uniform, and retrieval loses selectivity.
• If masking is required but omitted, a decoder can “look into the future,” violating causality.
• If sequence length grows very large, the $QK^\top$ matrix becomes the main memory and compute bottleneck.

The next section will address the first of these issues directly: why the raw dot product is divided by $\sqrt{d_k}$. For now, the key idea is that dot-product attention is not mysterious. It is a learned content-addressable lookup system implemented as matrix multiplication.

The visual below consolidates this flow. On the left, the equations isolate the three conceptual steps: define a pairwise compatibility score, vectorize all scores as $QK^\top$, and retrieve values using softmax-normalized weights. On the right, the matrix pipeline makes the same idea operational: rows of $Q$ meet rows of $K$ through $K^\top$, producing a score grid whose $(i,j)$ cell is exactly $q_i^\top k_j$.

The important thing to notice is the dense all-pairs communication pattern. Every query row has access to every key column before the softmax chooses how much value information to retrieve. That single pattern,

$$Z=\operatorname{softmax}(QK^\top)V,$$

is the algebraic heart of attention.

Having introduced dot products as a natural compatibility score between a query and a key, there is one more detail that looks like a harmless implementation trick but is actually crucial for stable learning: Transformer attention does not use $q_i^\top k_j$ directly. It uses a scaled dot product.

The raw score between query $q_i$ and key $k_j$ is

$$s_{ij}=q_i^\top k_j =\sum_{\ell=1}^{d_k} q_{i\ell}k_{j\ell}.$$

At first glance, this seems perfectly reasonable. If the query and key point in similar directions, the dot product is large; if they are unrelated or opposed, it is small or negative. But the magnitude of this score depends not only on semantic alignment. It also depends on the key/query dimension $d_k$. As $d_k$ grows, the dot product accumulates more random terms, so even unrelated vectors can produce scores with increasingly large variance.

A simple initialization-style calculation makes the issue visible. Suppose the components of $q_i$ and $k_j$ are independent, centered, and normalized:

$$\mathbb{E}[q_{i\ell}]=\mathbb{E}[k_{j\ell}]=0, \qquad \operatorname{Var}(q_{i\ell})=\operatorname{Var}(k_{j\ell})=1.$$

These assumptions are not meant to describe every trained Transformer exactly. They are a controlled approximation: before learning has shaped the representations too much, and under common normalization/initialization schemes, it is reasonable to ask what scale the scores would have if the components behaved like independent unit-variance random variables. The goal is to prevent the architecture itself from injecting an undesirable scale factor.

For one coordinate product, independence gives

$$\mathbb{E}[q_{i\ell}k_{j\ell}]=0,$$

and

$$\operatorname{Var}(q_{i\ell}k_{j\ell}) = \mathbb{E}[q_{i\ell}^2k_{j\ell}^2] - \mathbb{E}[q_{i\ell}k_{j\ell}]^2 =1.$$

So each coordinate contributes a unit-variance random term to the dot product. Since the full score sums $d_k$ such terms, the variance grows linearly:

$$\operatorname{Var}(s_{ij}) = \operatorname{Var}\!\left(\sum_{\ell=1}^{d_k}q_{i\ell}k_{j\ell}\right) = \sum_{\ell=1}^{d_k}\operatorname{Var}(q_{i\ell}k_{j\ell}) = d_k.$$

Equivalently, the standard deviation of the raw dot product grows like $\sqrt{d_k}$. This is the key point: increasing the representation dimension makes the scores larger in magnitude even when there is no stronger evidence of relevance.

That matters because attention does not use the scores directly; it passes them through a softmax. The softmax is sensitive to scale. If its inputs are small or moderate, it can assign a graded distribution over many keys. But if one logit is much larger than the others, the output becomes nearly one-hot:

$$\operatorname{softmax}(s)_j = \frac{e^{s_j}}{\sum_m e^{s_m}}.$$

Large score variance therefore pushes attention into a saturated regime. One key receives almost all the probability mass, the others receive almost none, and the gradients through the softmax become less informative. The model may still train, but optimization becomes unnecessarily brittle: early random score differences can dominate attention before the network has learned meaningful retrieval patterns.

The fix is to normalize the score by its typical standard deviation. Since

$$\operatorname{Std}(q_i^\top k_j)\approx \sqrt{d_k},$$

we compute attention using

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V.$$

This does not change the basic content-based retrieval story. Queries still compare themselves to keys, and values are still averaged according to the resulting attention weights. The scaling only keeps the logits in a numerically and statistically reasonable range as the key/query dimension changes.

A useful way to read the result is:

• Unscaled dot product: variance grows with $d_k$, so logits become large.
• Large logits: softmax becomes sharp or saturated.
• Scaled dot product: dividing by $\sqrt{d_k}$ restores roughly unit variance.
• Stable softmax: attention weights remain trainable and less dominated by accidental magnitude.

The visual below condenses this argument into two complementary views. On one side, the variance derivation tracks how the score $s_{ij}$ accumulates $d_k$ independent unit-variance products, ending in the highlighted conclusion $\operatorname{Var}(s_{ij})=d_k$. On the other side, the same phenomenon is shown operationally: unscaled scores become tall and uneven, producing a sharply peaked attention row, while scaled scores produce a smoother, more stable distribution.

The important takeaway is not that attention should always be diffuse. A trained Transformer can and often should place highly concentrated attention when the data calls for it. The point is that this concentration should be learned, not forced by the dimensionality of the dot product. Scaling by $\sqrt{d_k}$ makes dot-product attention behave consistently across dimensions, giving the softmax a well-conditioned set of logits to work with.

After controlling the scale of the dot products, the next issue is not how strongly one token should attend to another, but whether that link should exist at all. Attention, by default, is completely content-driven: every query compares itself with every key, and the softmax turns those comparisons into a probability distribution over all value vectors. That is powerful, but it is also too permissive. Some attention links are structurally invalid regardless of content.

The mechanism for enforcing these hard constraints is an attention mask. Starting from the scaled attention logits,

$$\frac{QK^\top}{\sqrt{d_k}},$$

we add a mask matrix $M$ before applying the row-wise softmax:

$$A=\operatorname{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}+M\right), \qquad Z=AV.$$

Here $M$ has the same row-column structure as the attention score matrix: rows correspond to query positions, columns correspond to key/value positions. The key idea is that masking happens at the logit level, before normalization. This matters because the softmax is sensitive to additive changes in logits: setting a logit to $-\infty$ makes its exponent exactly zero.

Concretely, for query position $i$ and key/value position $j$,

$$M_{ij}=0$$

means the link is allowed. The original scaled dot-product score is unchanged, so the model may assign attention mass to $v_j$ if the content match is strong. In contrast,

$$M_{ij}=-\infty$$

means the link is forbidden. Since

$$\exp(-\infty)=0,$$

the corresponding softmax weight becomes

$$a_{ij}=0.$$

Therefore $v_j$ contributes nothing to the output vector at query position $i$, no matter how compatible $q_i$ and $k_j$ might have been.

This is a hard structural constraint, not a learned preference. The model is not being encouraged to avoid certain links; it is made mathematically impossible for those links to carry information. In actual implementations, $-\infty$ is often represented by a very large negative number for numerical reasons, but the intended operation is the same: after softmax, the masked probability is zero or effectively zero.

The most important example is causal self-attention, used in autoregressive language modeling. When predicting token $t$, the model may use tokens at positions $u\le t$, but it must not look ahead to future positions $u>t$. Otherwise, training would leak the answer: the representation for an earlier token could directly depend on later tokens that should not yet be known during generation.

The causal mask is therefore

$$(M_{\mathrm{causal}})_{tu} = \begin{cases} 0, & u\le t,\\ -\infty, & u>t. \end{cases}$$

This gives a lower-triangular pattern, including the diagonal. Position $1$ can attend only to itself, position $2$ can attend to positions $1$ and $2$, and so on. The diagonal is usually allowed because the representation at a position may use the token currently being processed when computing hidden states for next-token prediction; what is forbidden is access to future positions.

The same additive-mask abstraction covers several different situations:

• Padding masks prevent real tokens from attending to padding tokens inserted for batching.
• Decoder self-attention masks enforce left-to-right causality.
• Autoregressive generation masks ensure that generation-time computation matches the information pattern assumed during training.
• Custom structural masks can restrict attention to windows, blocks, segments, or other task-specific connectivity patterns.

A subtle but important point is that masks operate independently of the learned parameters. The matrices $Q$, $K$, and $V$ are still produced by learned projections, and the model still decides among allowed links using content similarity. The mask simply defines the set of possible routes through which information may flow. In graph terms, attention learns edge weights, while the mask determines which edges are present.

The visual below condenses this into two complementary views. On the left, the mask appears exactly where it belongs mathematically: added to the scaled dot-product logits before the softmax. The two cases $M_{ij}=0$ and $M_{ij}=-\infty$ then become easy to interpret as “keep this candidate link” versus “force its attention weight to zero.”

On the right, the causal mask is represented as a triangular matrix. The allowed region contains zeros, while the forbidden future region contains $-\infty$. After the row-wise softmax, that forbidden upper triangle disappears from the attention matrix $A$: future value vectors cannot contribute to earlier query positions. This is the small algebraic trick that makes Transformer attention compatible with autoregressive sequence modeling.

After introducing masks, it is tempting to think of attention as a graph over sequence positions: some token positions may attend to others, and masking removes selected edges. But before we add masks—or positional encodings—there is a deeper fact hiding in plain sight: content-only self-attention does not know what a sequence order is. It sees an $n \times d_{\mathrm{model}}$ matrix $X$ as a collection of row vectors and computes interactions among those rows, but nothing in the vanilla attention formula says that row $3$ comes before row $4$, or that row $1$ is special because it is first.

Let $X \in \mathbb{R}^{n \times d_{\mathrm{model}}}$, where each row is a token representation. A single-head self-attention layer without positional information computes

$$Q = XW_Q,\qquad K = XW_K,\qquad V = XW_V,$$

and then

$$\operatorname{Attn}(X) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V.$$

The softmax is applied row-wise: each query row produces a probability distribution over all key rows, and then forms a weighted average of value rows. This is content-based retrieval: each token asks, “which other token vectors are relevant to me?” The important subtlety is that relevance is computed only through dot products of learned projections. There is no term involving the integer index $i$, no sinusoidal or learned positional vector $P_i$, and no causal or padding mask $M$ that distinguishes allowed from disallowed positions.

Now consider a permutation matrix $\Pi \in \mathcal{P}_n$. Multiplying $X$ on the left by $\Pi$ simply reorders the rows of $X$. For example, if $X$ contains token vectors in one order, then $\Pi X$ contains the same token vectors in a different order. The theorem says that self-attention commutes with this reordering:

$$\operatorname{Attn}(\Pi X) = \Pi \operatorname{Attn}(X).$$

That property is called permutation equivariance. It is not invariance: the output does change when the input is permuted. But it changes in exactly the same way—the output rows are permuted by the same $\Pi$. In other words, content-only self-attention treats the input as an unordered set of token vectors, while still returning one output vector per input token.

The algebra is short but instructive. If we permute the input rows, the projected queries, keys, and values become

$$Q' = \Pi XW_Q = \Pi Q,\qquad K' = \Pi XW_K = \Pi K,\qquad V' = \Pi XW_V = \Pi V.$$

The new attention score matrix is

$$Q'{K'}^\top = (\Pi Q)(\Pi K)^\top = \Pi QK^\top \Pi^\top.$$

So the score matrix is not arbitrary; it is the original score matrix with both rows and columns permuted. Rows are permuted because the queries have been reordered, and columns are permuted because the keys have been reordered. The scaling by $\sqrt{d_k}$ does not affect this symmetry.

The only slightly delicate step is the row-wise softmax. For any score matrix $S$,

$$\operatorname{softmax}(\Pi S \Pi^\top) = \Pi \operatorname{softmax}(S)\Pi^\top.$$

This holds because permuting a row before applying softmax simply permutes the resulting probabilities in the same way, and permuting the collection of rows also just reorders the row-wise outputs. Applying this to

$$S = \frac{QK^\top}{\sqrt{d_k}},$$

we get

$$\operatorname{softmax}\!\left(\frac{Q'{K'}^\top}{\sqrt{d_k}}\right) = \Pi \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) \Pi^\top.$$

Multiplying by the permuted values $V' = \Pi V$ cancels the inner permutation:

$$\operatorname{softmax}\!\left(\frac{Q'{K'}^\top}{\sqrt{d_k}}\right)V' = \Pi \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right) \Pi^\top \Pi V = \Pi \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V.$$

Substituting back $Q=XW_Q$, $K=XW_K$, and $V=XW_V$, the theorem is exactly

$$\operatorname{softmax}\!\left(\frac{(\Pi X W_Q)(\Pi X W_K)^\top}{\sqrt{d_k}}\right)(\Pi X W_V) = \Pi\operatorname{softmax}\!\left(\frac{(XW_Q)(XW_K)^\top}{\sqrt{d_k}}\right)(XW_V).$$

This result matters because language is not merely a bag of token embeddings. The sentences “dog bites man” and “man bites dog” contain the same token set but mean different things. A Transformer without positional information cannot distinguish these two sequences by order alone. If the token embeddings are identical and only their rows are rearranged, the layer can only rearrange its outputs correspondingly. It has no intrinsic mechanism for representing “before,” “after,” “nearby,” or “first.”

There are also useful boundary cases to keep in mind. The equivariance statement assumes no positional encoding $P$, no mask $M$, and no position-dependent biases. Adding absolute positional embeddings breaks the symmetry because row $i$ receives information tied to index $i$. Adding a causal mask also breaks full permutation equivariance because the mask privileges the left-to-right order. By contrast, operations applied identically to every row—such as a shared feed-forward network, residual addition, or layer normalization over features—typically preserve permutation equivariance. The symmetry is broken only when the model is given some information that distinguishes positions.

The visual below compactly summarizes the theorem as a commuting diagram: one path permutes the input rows first and then applies self-attention, while the other applies self-attention first and then permutes the output rows. The equality says these paths arrive at the same result. That is the operational meaning of permutation equivariance.

The equation in the theorem box is the algebraic version of the same story. The orange $\Pi$ terms track row permutations, the blue softmax block tracks how attention weights reorder consistently, and the green value block reminds us that the final weighted sums are attached to the same permuted rows. The important takeaway is simple but profound: without positions or masks, self-attention is a powerful content-retrieval mechanism, but not yet a sequence model in the ordered sense.

The equivariance result is useful precisely because it tells us what self-attention cannot know by itself. If the inputs are just token embeddings, self-attention treats the sequence as a set of content vectors with indices attached only externally. It can route information based on what appears, but not intrinsically on where it appears. That is elegant from a symmetry perspective, but disastrous for language, programs, music, genomes, and essentially every sequence domain where order changes meaning.

For example, the two strings “the dog bit the man” and “the man bit the dog” contain almost the same multiset of token identities. A content-only self-attention layer can produce correspondingly permuted representations, but it has no built-in reason to distinguish subject position from object position. The problem is not that attention is weak; the problem is that attention is too symmetric. We must deliberately break permutation symmetry by injecting positional information.

The standard Transformer does this by replacing each token embedding $e_i$ with a position-aware input representation such as

$$x_i = e_i + p_i,$$

where $p_i$ is a vector associated with position $i$. Now the attention mechanism no longer receives just “the embedding for this word”; it receives “the embedding for this word at this location.” The dot products used to compute attention can depend on token identity, position, and interactions between the two. In other words, attention remains content-based retrieval, but the content being retrieved has been enriched with location.

There are several ways to choose the positional signal. The original Transformer used sinusoidal positional encodings, where each coordinate varies periodically with position at a different frequency. Learned absolute embeddings are also common: the model simply learns a table $p_1, p_2, \dots, p_n$. More recent architectures often use relative position biases or rotary positional embeddings, which modify the attention scores or query/key geometry so that the model reasons more directly about offsets like “three tokens ago” rather than absolute coordinates like “position 57.”

These choices differ in what generalization they encourage:

• Absolute learned positions are simple and expressive, but may extrapolate poorly beyond the training context length.
• Sinusoidal positions provide a fixed geometric structure and make some extrapolation possible, though not automatically robust.
• Relative position methods often fit language modeling well because many linguistic dependencies are local or distance-sensitive.
• Rotary embeddings encode relative displacement through rotations in query-key space, preserving the dot-product attention framework while adding position-dependent phase structure.

A subtle point is that masking is not a full substitute for positional encoding. A causal mask does introduce an ordering constraint: token $i$ cannot attend to future tokens $j > i$. But without positional information, the model still has limited ability to distinguish different permutations of the visible prefix. The mask says “you may look backward,” but it does not fully tell the model which earlier token was first, second, or adjacent in a content-independent way. For sequence modeling, causality and position solve different problems: causality prevents information leakage, while positional encoding gives the model a coordinate system.

This also explains why positional information matters even in encoder-only models, where there is no causal mask. In a bidirectional encoder, every token can attend to every other token. Without positions, the representation of a sentence is equivariant to arbitrary reordering. A downstream classifier might collapse those token representations into something nearly permutation-invariant, making word order even harder to recover. Positional encodings give the encoder a way to represent syntactic roles, local neighborhoods, phrase boundaries, and long-range dependencies as structured relations rather than unordered co-occurrences.

The visual summary for this idea should be read as a symmetry-breaking story: content-only self-attention preserves permutation structure, while adding positional signals turns a bag-like collection of embeddings into an ordered sequence. Once token and position information are combined, attention can still retrieve by similarity, but similarity is now computed in a space where “same word in a different place” can mean something different.

This sets up the next architectural refinement. After we give the model a notion of position, a single attention operation still represents only one retrieval pattern at a time. In practice, different relationships matter simultaneously: nearby syntax, long-range agreement, delimiter matching, coreference, copying, and positional offsets. Multi-head attention will let the model learn several such retrieval subspaces in parallel.

Before assembling a full Transformer block, it is worth pausing on the attention mechanism itself. A single scaled dot-product attention layer already gives us a powerful content-addressable retrieval operation: each token forms a query, compares it against all keys, and uses the resulting weights to average the corresponding values. But one attention distribution is still only one way of asking, “What information should this token retrieve from the sequence?”

The central idea of multi-head attention is that a token may need to retrieve several different kinds of evidence at once. In language, for example, a word might need nearby syntactic context, a long-range subject for agreement, a previous mention for coreference, and a delimiter or boundary token for structure. These are not necessarily well represented by a single similarity function over one query-key space. Multi-head attention addresses this by running several attention mechanisms in parallel, each with its own learned projections.

Given an input sequence representation $X$, each head $r \in \{1,\ldots,h\}$ learns separate projection matrices

$$W_Q^{(r)}, \qquad W_K^{(r)}, \qquad W_V^{(r)}.$$

These produce head-specific queries, keys, and values:

$$Q^{(r)} = XW_Q^{(r)}, \qquad K^{(r)} = XW_K^{(r)}, \qquad V^{(r)} = XW_V^{(r)}.$$

The head then performs the same scaled dot-product attention operation we have already developed:

$$\operatorname{head}_r = \operatorname{softmax}\!\left( \frac{(XW_Q^{(r)})(XW_K^{(r)})^\top}{\sqrt{d_k}} + M \right) (XW_V^{(r)}).$$

The mask $M$ plays the same role as before: it can forbid attention to certain positions, such as future tokens in causal decoding or padding tokens in batched training. The scaling by $\sqrt{d_k}$ also remains essential, because each head computes dot products in its own key/query dimension $d_k$. Without scaling, the logits can grow too large in magnitude as $d_k$ increases, causing the softmax to become overly peaked and gradients to become less useful.

The important change is not the formula inside one head; it is the fact that each head has its own learned retrieval subspace. One head might learn projections where query-key similarity emphasizes syntactic adjacency. Another might emphasize semantic similarity. Another might specialize in positional or delimiter-like patterns. This specialization is not manually assigned; it emerges because the model can reduce training loss by distributing different retrieval behaviors across heads.

After computing all heads in parallel, their outputs are concatenated:

$$\operatorname{Concat}(\operatorname{head}_1,\ldots,\operatorname{head}_h).$$

This concatenated representation contains multiple retrieved views of the sequence at each token position. A final learned output projection $W_O$ then mixes these views back into the model dimension:

$$\operatorname{MHA}(X) = \operatorname{Concat}(\operatorname{head}_1,\ldots,\operatorname{head}_h)W_O.$$

This final projection is easy to underestimate. Concatenation alone would merely place the heads side by side. The output matrix $W_O$ lets the model form learned combinations across heads: it can amplify, suppress, or blend information retrieved by different attention patterns. In other words, multi-head attention is not just “several attentions in parallel”; it is parallel retrieval followed by a learned recombination step.

A common implementation choice is to keep the total compute roughly comparable to a single large attention layer by splitting the model dimension across heads. For example, if the model width is $d_{\text{model}}$ and there are $h$ heads, one often uses

$$d_k = d_v = \frac{d_{\text{model}}}{h}.$$

Then each head is narrower, but there are more of them. This gives the model multiple attention patterns without multiplying the representation size before the output projection. The trade-off is that each individual head has lower dimensional capacity, while the collection of heads has greater diversity in possible retrieval behavior.

There are also subtle failure modes. Heads are not guaranteed to become neatly interpretable modules such as “syntax head” or “coreference head.” Some heads may be redundant, diffuse, or useful only in combination with others. In practice, attention patterns are informative but not always faithful explanations of model behavior. Still, the architectural bias matters: by giving the model several independent query-key-value projections, multi-head attention makes it easier to represent multiple relational structures simultaneously.

The visual below compactly organizes this computation from left to right. The same input $X$ fans out into several parallel lanes, one per head. Each lane applies its own $Q$, $K$, and $V$ projections, performs masked scaled dot-product attention, and emits a head-specific retrieved representation. The different colors emphasize that these heads are not copies of one another; they are separate learned retrieval mechanisms operating in distinct subspaces.

On the right side, the heads are gathered by concatenation and passed through $W_O$, which mixes them back into a single output representation $\operatorname{MHA}(X)$. This is the key structural pattern to remember before moving to the rest of the Transformer block: parallel attention heads create multiple retrieved views, and the output projection integrates those views into the next token representation.

After multi-head attention has gathered information from different positions and different representation subspaces, the Transformer still needs a way to compute new features from the resulting token vectors. Attention is excellent at routing and mixing information across the sequence, but the weighted sums it produces are still largely linear combinations of value vectors. To make each token representation more expressive, every Transformer block follows attention with a position-wise feed-forward network.

The phrase position-wise is important. Suppose the block input is a matrix

$$X \in \mathbb{R}^{n \times d_{\mathrm{model}}},$$

where row $X_i$ is the representation of token position $i$, already combining token identity and positional information. After multi-head attention, each row has had the opportunity to receive information from other rows. The feed-forward layer then applies the same nonlinear map to each row independently:

$$\operatorname{FFN}(X)_i = \phi(X_i W_1 + b_1) W_2 + b_2, \quad i=1,\ldots,n.$$

Here $W_1,b_1,W_2,b_2$ are shared across all positions, and $\phi$ is a pointwise nonlinearity such as ReLU or GELU. In modern Transformers, GELU-like activations are common, but the architectural idea is not tied to one particular choice.

A useful mental model is that attention answers the question: Which other positions should this token read from? The feed-forward network answers a different question: Given the information now stored in this token vector, how should we transform its features? These are complementary operations:

• Attention: mixes information between rows of $X$.
• FFN: transforms information within each row of $X$.
• Parameter sharing: the same transformation is reused at every sequence position.

This independence means there is no communication between positions inside the FFN itself. If position $i$ affects position $j$, that influence must have already been routed through attention or must happen in a later attention layer. The FFN is therefore not a replacement for attention; it is the nonlinear feature processor that acts after attention has assembled a useful local representation at each position.

The standard Transformer FFN has a characteristic expand-and-compress shape:

$$d_{\mathrm{model}} \rightarrow d_{\mathrm{ff}} \rightarrow d_{\mathrm{model}}, \quad d_{\mathrm{ff}} > d_{\mathrm{model}}.$$

The first linear map projects each token vector into a wider hidden space of dimension $d_{\mathrm{ff}}$. The activation $\phi$ introduces nonlinearity, allowing the model to form feature interactions that cannot be represented by attention’s weighted averaging alone. The second linear map compresses the representation back to $d_{\mathrm{model}}$, so the output can be passed cleanly to the next sublayer in the block.

This expansion matters. If the FFN were only a single linear map from $d_{\mathrm{model}}$ to $d_{\mathrm{model}}$, then it would add limited expressive power, especially when surrounded by other linear projections. The intermediate width $d_{\mathrm{ff}}$ gives the model a larger workspace for computing token-wise features: detecting patterns, gating dimensions, composing semantic attributes, or re-encoding information gathered from attention. In many Transformer configurations, $d_{\mathrm{ff}}$ is several times larger than $d_{\mathrm{model}}$, making the FFN a major contributor to both parameter count and computation.

There is a subtle but important symmetry here. Because the same FFN parameters are applied to every row, the operation is shared over sequence length. The model does not learn one feed-forward map for the first token, another for the second token, and so on. Instead, positional differences are represented in the input vectors themselves, while the transformation rule remains the same everywhere. This sharing is one reason Transformers can process variable-length sequences: the FFN does not depend on a fixed sequence length $n$.

Equivalently, we can view the FFN as a tiny multilayer perceptron applied in parallel to all token positions. In matrix form, ignoring broadcasting details for the biases, this is

$$\operatorname{FFN}(X) = \phi(XW_1 + b_1)W_2 + b_2,$$

but this compact expression can hide the crucial fact that row $i$ is transformed without directly reading row $j$. The matrix notation is efficient; the row-wise interpretation is the architectural insight.

The visual below condenses this idea into a left-to-right pipeline. Each row vector $X_1,\ldots,X_n$ enters an identical two-layer nonlinear transformation: expansion by $W_1,b_1$, activation by $\phi$, and compression by $W_2,b_2$. The parallel lanes emphasize that there are no arrows between positions inside the FFN.

The shared-weight annotation is just as important as the lanes themselves. It reminds us that the FFN is independent across rows but not separately parameterized across rows. The same learned map is reused at every position, turning attention’s cross-token communication into richer per-token features before the block moves on to residual connections and normalization.

After adding the position-wise feed-forward network, we now have the two computational ingredients that make up a Transformer block: multi-head attention for token-to-token interaction, and an MLP/FFN for per-token nonlinear transformation. But simply stacking these transformations naively is usually unstable. Deep networks need a way to preserve information, keep gradients healthy, and prevent activations from drifting into poorly scaled regimes.

This is where the Transformer block becomes more than “attention followed by an MLP.” Each major sublayer is wrapped with three stabilizing mechanisms:

• a residual connection, which adds the sublayer input back to its output;
• dropout, which regularizes the sublayer output during training;
• layer normalization, which rescales each token representation to a controlled feature distribution.

In the original Transformer formulation, these are arranged in a post-normalization pattern. For an input sequence representation $X^{(\ell)} \in \mathbb{R}^{n \times d_{\text{model}}}$, the attention sublayer produces an intermediate representation

$$Z=\operatorname{LN}\!\left( X^{(\ell)} + \operatorname{Dropout}(\operatorname{MHA}(X^{(\ell)})) \right).$$

Then the feed-forward sublayer is applied with the same wrapper:

$$X^{(\ell+1)} = \operatorname{LN}\!\left( Z + \operatorname{Dropout}(\operatorname{FFN}(Z)) \right).$$

The residual addition is not just a convenience. It gives the block a short identity route through depth. If the attention or feed-forward transformation is initially unhelpful, the model can still pass forward something close to the original representation. This matters because deep Transformers are trained by gradient descent: without residual paths, every layer would have to learn both how to preserve useful information and how to modify it. With residual paths, a sublayer can instead learn a correction or update to the current representation.

A useful way to think about one wrapped sublayer is

$$\text{new representation} \approx \text{old representation} + \text{learned update}.$$

Attention contributes a context-dependent update, while the FFN contributes a token-wise nonlinear update. The residual path ensures that these updates are added to an existing representation rather than replacing it entirely. This “incremental refinement” viewpoint is one reason very deep residual architectures are trainable.

Dropout is placed on the sublayer output before the residual addition. During training, this randomly removes parts of the proposed update. The identity path remains intact, so dropout regularizes the transformation without fully corrupting the information stream. In other words, the model is discouraged from relying too heavily on any one attention head, hidden feature, or feed-forward activation, while still preserving a stable baseline signal through the residual branch.

Layer normalization then controls the scale of the resulting token representations. For each token independently, layer normalization computes statistics across the feature dimension and normalizes the vector. Abstractly, for a token vector $x \in \mathbb{R}^{d}$,

$$\operatorname{LN}(x) = \gamma \odot \frac{x-\mu}{\sqrt{\sigma^2+\epsilon}}+\beta,$$

where $\mu$ and $\sigma^2$ are computed over the features of that token, and $\gamma,\beta$ are learned scale and shift parameters. This differs from batch normalization: the normalization does not depend on other examples in the batch or other positions in the sequence. That property is especially important for variable-length sequence models and autoregressive decoding.

There is a subtle but important architectural variation here. The equations above describe post-normalization, where normalization happens after the residual addition. Many modern large language models instead use pre-normalization, where the input is normalized before the attention or FFN sublayer, and the residual addition happens afterward. The ingredients are the same, but the order changes:

• Post-norm: sublayer $\rightarrow$ dropout $\rightarrow$ residual add $\rightarrow$ layer norm.
• Pre-norm: layer norm $\rightarrow$ sublayer $\rightarrow$ dropout $\rightarrow$ residual add.

Pre-normalization often improves optimization stability for very deep Transformers because gradients can flow more directly through the residual stream. Post-normalization was used in the original Transformer and remains conceptually clean, but it can become harder to train as depth increases unless additional care is taken with initialization, learning-rate schedules, or normalization variants.

The visual below should now read as a compact assembly diagram for the post-normalization Transformer block. The main vertical path applies MHA and then FFN, while the curved bypass arrows represent the residual identity routes. Each learned update passes through Dropout, is added back to the incoming representation, and is then stabilized by LN.

The key idea is that the block is not a plain stack of transformations. It is a repeated pattern of propose an update, regularize it, add it to the residual stream, normalize the result. That pattern is what lets attention and feed-forward layers be composed deeply enough to form the backbone of modern Transformer models.

With residual connections and layer normalization in place, a Transformer block has a stable way to transform and refine representations. But there is still a surprisingly fundamental problem: self-attention itself does not know what order the tokens came in.

The reason is that vanilla self-attention is a content-based retrieval mechanism. Each token produces a query, key, and value; attention compares queries to keys and mixes values according to similarity. If we permute the rows of the input matrix $X$, then the queries, keys, values, attention weights, and outputs are permuted in the same way. In other words, self-attention is permutation equivariant: reordering the input sequence merely reorders the output sequence. That is a useful symmetry for sets, but language is not a set. The sentences “dog bites man” and “man bites dog” contain the same words, but their meanings are not interchangeable.

So the Transformer must break this symmetry deliberately. It does not do so by recurrence, where position is implicit in the order of computation, nor by convolution, where locality is built into the kernel geometry. Instead, it injects position information into an otherwise order-agnostic attention mechanism. Broadly, there are two families of solutions:

• Absolute positional information: attach a representation of position $i$ directly to the token representation.
• Relative positional information: modify attention scores so that position $i$ attending to position $j$ depends on their relative offset $i-j$.

The original Transformer used absolute positional encodings. If $e(x_i)\in\mathbb{R}^{d_{\mathrm{model}}}$ is the token embedding for token $x_i$, then we add a position vector $p_i$ before the first attention layer:

$$X= \begin{bmatrix} e(x_1)+p_1 \\ \cdots \\ e(x_n)+p_n \end{bmatrix} \in\mathbb{R}^{n\times d_{\mathrm{model}}}.$$

This changes the meaning of the row representation. A row no longer says only “this is the word bank”; it says something closer to “this is the word bank at position $i$.” Once that information is inside the representation, the attention mechanism can learn position-sensitive behavior through ordinary dot products. For example, a head may learn that a token near the beginning of a sentence behaves differently from the same token near the end, or that certain syntactic patterns depend on approximate position.

There are two common variants of absolute positions. In learned absolute positions, each $p_i$ is a trainable vector, just like a word embedding. This is simple and flexible, but it ties the model to the range of positions seen during training unless special care is taken. In fixed sinusoidal positions, $p_i$ is a deterministic function of $i$, using sine and cosine waves at multiple frequencies. The motivation is that different dimensions encode position at different resolutions, and relative offsets can be expressed through linear relationships among these periodic features. Fixed encodings are not learned from data, but they provide a structured notion of position that can extrapolate more gracefully in some settings.

Absolute positions are intuitive, but they have a subtle limitation: they identify where a token is in the sequence, not directly how far apart two tokens are. Many linguistic and sequential patterns are naturally relative. A word may care about the previous token, the next token, the nearest verb, or another symbol three positions back. For these cases, it is often more natural to inject order into the attention score itself.

In standard scaled dot-product attention, the score from query position $i$ to key position $j$ is

$$s_{ij}=\frac{q_i^\top k_j}{\sqrt{d_k}}.$$

Relative positional methods modify this idea so that the score also depends on the displacement between the two positions:

$$s_{ij} = \frac{q_i^\top k_j}{\sqrt{d_k}} \quad\longrightarrow\quad s_{ij}\text{ also depends on }i-j.$$

This is a different way of breaking permutation symmetry. Instead of saying “token $x_i$ carries position vector $p_i$,” the model says “when position $i$ attends to position $j$, the interaction depends on their distance and direction.” The sign of $i-j$ matters: attending three tokens to the left is not the same as attending three tokens to the right. In practice, this can be implemented using additive biases, relative key/value embeddings, rotary transformations, or other mechanisms, but the conceptual move is the same: make attention pairwise position-aware.

The distinction matters because each approach gives the model a different inductive bias. Absolute positions are simple and global: every token knows its address. Relative positions are relational: every attention edge knows its offset. Absolute positions can be enough for many tasks, especially when sequence lengths are bounded and consistent. Relative schemes often work better when patterns depend on local displacement, when length extrapolation matters, or when the model benefits from treating “nearby” and “far away” interactions differently regardless of absolute location.

The visual below consolidates these two routes into the same conceptual frame. On the left, order enters before attention: token embeddings $e(x_i)$ are combined with position vectors $p_i$, producing the matrix $X\in\mathbb{R}^{n\times d_{\mathrm{model}}}$. On the right, order enters inside attention: the score grid is no longer determined only by content similarity $q_i^\top k_j$, but also by diagonal bands corresponding to relative offsets $i-j$.

The key takeaway is that Transformers do not obtain order “for free.” Self-attention gives them flexible content-based communication, residual paths preserve and refine representations, and layer normalization stabilizes the computation—but positional information is what turns a permutation-equivariant set processor into a sequence model. Order enters either through the representation $p_i$, or through the attention interaction $s_{ij}$, rather than through recurrence.

After adding positional information, we have fixed one major ambiguity of self-attention: a token representation can now know where it sits. But in the decoder there is a second, equally important constraint: it must not know what comes next. If a model is trained to predict the next token while its internal representation can already attend to future tokens, the learning problem becomes contaminated by future-token leakage. The model may appear to achieve excellent training loss, but it is solving the wrong conditional distribution.

The autoregressive modeling assumption is that a sequence distribution factors as

$$p_\theta(x_1,\ldots,x_n) = \prod_{t=1}^n p_\theta(x_t\mid x_{<t}).$$

So when the model produces the conditional distribution for position $t$, its computation may depend on $x_1,\ldots,x_{t-1}$, but not on $x_t,x_{t+1},\ldots,x_n$ depending on the exact indexing convention. Equivalently, if we feed the decoder a prefix ending at position $t$, the representation at that position may depend only on the prefix. In the common teacher-forcing setup, inputs are shifted so that the row at one position is used to predict the next token; the same structural requirement remains: no representation used for prediction may incorporate information from tokens to its right.

Causal masking enforces this directly inside self-attention. Recall that attention weights are computed from logits of the form

$$\frac{q_t^\top k_u}{\sqrt{d_k}} + M_{tu},$$

followed by a row-wise softmax over key/value positions $u$. In decoder self-attention, the additive mask is

$$(M_{\mathrm{causal}})_{tu}=0\ \text{for }u\le t,\qquad (M_{\mathrm{causal}})_{tu}=-\infty\ \text{for }u>t.$$

Thus, for query position $t$, keys and values at positions $u>t$ receive logit $-\infty$. After softmax, their probability mass is exactly zero:

$$a_{tu}=0\qquad \text{for all }u>t.$$

This is the key mechanism. The mask does not merely discourage looking ahead; in the mathematical idealization, it removes those edges from the computation graph.

Now consider a full stack of masked Transformer decoder blocks. The input at position $i$ is

$$X_i^{(0)}=e(x_i)+p_i,$$

where $e(x_i)$ is the token embedding and $p_i$ is positional information. The theorem says that for every layer $\ell$ and every position $t$, the row $X_t^{(\ell)}$ is a function only of

$$(x_1,\ldots,x_t) \quad\text{and}\quad (p_1,\ldots,p_t),$$

not of any future token $x_u$ with $u>t$. This is what makes decoder-only Transformers valid autoregressive models: their internal states respect the same left-to-right conditional structure as the probability distribution they are trained to represent.

The proof is a simple but important induction over layers. At layer $0$, the row $X_t^{(0)}$ depends only on $x_t$ and $p_t$, so it certainly does not depend on future tokens. Assume that at layer $\ell$, every row $X_u^{(\ell)}$ depends only on tokens up to position $u$. When computing masked self-attention for row $t$ at the next layer, the query at $t$ can attend only to rows $u\le t$. By the induction hypothesis, each such row $X_u^{(\ell)}$ depends only on tokens $x_1,\ldots,x_u$, and since $u\le t$, all of those are contained in $x_1,\ldots,x_t$. Therefore the attention output at position $t$ depends only on the prefix up to $t$.

The remaining parts of the Transformer block preserve this property. The feed-forward network is applied position-wise, so it cannot mix information across sequence positions. Residual connections add together quantities from the same row. Layer normalization, in the usual Transformer form, normalizes across feature dimensions within a row, not across time positions. Therefore residual-plus-normalization also cannot introduce dependence on future tokens. The induction closes: every layer preserves the causal dependency structure.

There are a few subtle assumptions hiding inside this clean theorem:

• The mask must be applied before the softmax, and future logits must be treated as truly impossible, ideally $-\infty$, not merely a small penalty.
• Normalization must be row-wise. A normalization operation that aggregates statistics across time could leak future information.
• Positional information must not encode future token content. Absolute position vectors $p_i$ are fine because they reveal where a token is, not what future tokens are.
• Relative positional biases are also fine as long as they respect the causal mask and do not reintroduce attention to $u>t$.

In real implementations, $-\infty$ is often approximated by a very large negative number. This is usually safe in floating-point arithmetic, but conceptually the theorem relies on the masked attention weights being exactly zero for future positions. Bugs in masking, off-by-one indexing errors, or applying the mask with the wrong tensor shape can produce silent leakage. Such leakage is especially dangerous because training loss may improve while generation quality or evaluation validity becomes compromised.

The practical payoff is that we can train on full sequences in parallel while still modeling left-to-right conditionals. During training, the model sees the entire length-$n$ sequence as a tensor, but the causal mask ensures that the representation used for each prediction only has access to the appropriate prefix. This is the core reason Transformer decoders avoid the sequential computation bottleneck of RNNs while still parameterizing distributions like

$$p_\theta(x_t\mid x_{<t}).$$

The visual below compactly summarizes the theorem’s two perspectives. On the left, the causal mask is a lower-triangular attention pattern: positions may attend backward and to themselves, but the strict upper triangle is removed. A highlighted query row $t$ makes the statement concrete: all entries with $u>t$ are blocked, which corresponds exactly to $a_{tu}=0$.

On the right, the same idea is lifted from one attention matrix to an entire stack of decoder blocks. Information from positions $1,\ldots,t$ can flow upward into $X_t^{(\ell)}$, while arrows from future positions are stopped before reaching it. That is the theorem in computational-graph form: after any number of masked self-attention layers, the row used for autoregressive prediction still depends only on the prefix, giving next-token modeling with no leakage.

The causal masking theorem gives us more than a correctness condition for language modeling. It tells us that attention is a programmable dependency pattern: by changing which positions are allowed to communicate, we change the computational graph of the model without changing the basic attention mechanism itself.

That is the key architectural leap. A Transformer is not “an attention layer” repeated many times in the abstract. It is a stack of modules where each module answers three separate questions:

• What information is available? Full sequence, prefix only, or another sequence entirely?
• How is information selected? By content-based attention weights.
• How is each token state updated? Through residual connections, normalization, and position-wise nonlinear transformations.

The previous result focused on the first question for autoregressive models. If token $i$ is only allowed to attend to positions $\leq i$, then its representation cannot depend on future tokens. That makes next-token prediction valid: the model cannot “cheat” by reading the answer from the right-hand side of the sequence.

But causal masking is only one possible attention topology. The same attention operation can support very different modeling regimes depending on the allowed communication pattern:

• In an encoder, every token may attend to every other token. This is useful when the whole input is known in advance, as in classification, retrieval, or the source side of translation.
• In a decoder, each token may attend only to itself and earlier tokens. This is what makes autoregressive generation well-defined.
• In cross-attention, decoder states attend to encoder states. This lets a generated target sequence condition on a separate input sequence.

A useful way to think about this is that attention defines a soft message-passing graph over token representations. The mask determines which edges exist; the attention scores determine how strongly each available edge is used. The model does not merely copy from neighboring positions. It learns, at every layer and every head, which previous or external states are relevant to the current computation.

This separation matters because many Transformer properties come from structural constraints, not from learned parameters alone. If we remove the causal mask from a language-model decoder during training, the model may achieve an artificially low loss by using future tokens. If we impose a causal mask inside an encoder, we unnecessarily prevent tokens from using right context. If we omit positional information, self-attention remains largely insensitive to token order except through whatever asymmetries are introduced elsewhere. The architecture works because these design choices are aligned with the task.

There is also a computational reason to keep these pieces modular. During training, even causal self-attention can be evaluated in parallel across all positions because the mask is known in advance. The model computes all token representations simultaneously while enforcing the same dependency pattern that will hold during generation. During decoding, however, generation is sequential: after predicting one token, the model appends it to the prefix and runs the next step. This is why training and inference have different bottlenecks even though they use the same learned layers.

The next architectural step is therefore to assemble these attention patterns into reusable blocks. Each block takes a sequence of hidden states, routes information through an attention sublayer, applies a token-wise feed-forward transformation, and preserves trainability through residual and normalization structure. The distinction between encoder, decoder, and encoder-decoder models is mostly a distinction in which attention sublayers are present and what they are allowed to see.

The visual below serves as a compact transition from the theorem to the architecture. Instead of treating masking as a small implementation detail, it places masking and attention access patterns at the center of the design. Once that picture is clear, encoder self-attention, decoder causal self-attention, and cross-attention become variations of the same underlying operation rather than separate mechanisms.

It is worth carrying this mental model forward: attention computes content-based retrieval; masks define legal information flow; Transformer blocks package that retrieval into stable, trainable layers. With that in place, we can now build the encoder, decoder, and encoder-decoder Transformer architectures systematically.

A useful way to organize the Transformer design space is to stop thinking in terms of “different architectures” and instead ask two more primitive questions: which tokens are allowed to attend to which other tokens, and where do the queries, keys, and values come from? Once causal masking is understood, the distinction between encoder-only, decoder-only, and encoder–decoder Transformers becomes much less mysterious. They are mostly the same attention operation, wired with different masks and different source streams.

Recall the core attention computation:

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right)V.$$

The matrix $QK^\top$ scores how much each query position wants to retrieve information from each key position. The mask $M$ then changes which retrievals are legal. Typically, allowed entries receive $0$, while forbidden entries receive a very negative number, effectively making their softmax probability zero. So the mask is not a minor implementation detail; it defines the information flow graph of the model.

In an encoder-only Transformer, all positions in the input sequence $x_{1:n}$ can usually attend to all other non-padding positions. There is no causal restriction because the goal is not to generate the sequence left-to-right. Instead, the model builds a contextual representation $Z$, where each token representation may depend on tokens both to its left and to its right. This is appropriate for tasks like classification, retrieval, tagging, and masked-token prediction, where the entire input is available at once.

The subtle assumption here is that bidirectional context is legal. If the downstream task requires predicting the future from the past, an encoder-only model would leak information unless we carefully modify the objective or mask. But when the whole input is genuinely observed, full self-attention is a strength: every token can be interpreted in light of the complete sequence.

In a decoder-only Transformer, the same sequence $x_{1:n}$ is treated as a language-modeling sequence. Position $t$ is trained to predict the next or current token using only earlier tokens. The model therefore uses causal self-attention, meaning token $t$ may attend to positions $\le t$, but not to positions $>t$. This gives the familiar autoregressive factorization:

$$p_\theta(x_{1:n}) = \prod_{t=1}^n p_\theta(x_t \mid x_{<t}).$$

This architecture is natural for open-ended generation because the model’s training-time information pattern matches its test-time use: when generating token $x_t$, the future tokens do not yet exist. A failure mode appears when this alignment is broken. If a decoder accidentally receives unmasked future tokens during training, it can learn a shortcut that disappears at inference time. Causal masking is therefore what makes parallel training compatible with left-to-right generation.

An encoder–decoder Transformer separates the problem into two streams. The encoder first reads the source sequence $x_{1:n}$ and produces contextual representations $Z$. The decoder then generates a target sequence $y_{1:m}$ autoregressively. Within the decoder, causal self-attention ensures that position $t$ can only depend on $y_{<t}$. But after that, the decoder also performs cross-attention over the encoder output. This gives the conditional prediction form

$$p_\theta(y_{1:m}\mid x_{1:n}) = \prod_{t=1}^m p_\theta(y_t \mid y_{<t},x_{1:n}).$$

Cross-attention is not a new mathematical primitive. It is the same content-based retrieval operation, but with a different source for $Q,K,V$. The decoder state supplies the queries $Q$: “given what I have generated so far, what source information do I need?” The encoder output supplies the keys and values $K,V$: “here are the source-side memories available for retrieval.” In compact form,

$$Q \leftarrow \text{decoder states}, \qquad K,V \leftarrow Z.$$

This distinction matters because it separates two kinds of dependency. Decoder self-attention models dependencies among generated target tokens, while cross-attention conditions those target tokens on the input. For translation, summarization, speech recognition, and other sequence-to-sequence tasks, this is exactly what we want: the output should be fluent in its own sequence while remaining grounded in the source.

The three families can therefore be compared by a small set of choices:

• Encoder-only: bidirectional self-attention over an observed input; produces contextual representations.
• Decoder-only: causal self-attention over a generated sequence; models $p_\theta(x_t \mid x_{<t})$.
• Encoder–decoder: causal decoder self-attention plus cross-attention to an encoded source; models $p_\theta(y_t \mid y_{<t},x_{1:n})$.

The visual below condenses this taxonomy into a table: the rows are not fundamentally different attention formulas, but different choices about masks and streams. The important thing to notice is that “self-attention” means $Q,K,V$ come from the same sequence, whereas “cross-attention” means the decoder provides $Q$ and the encoder provides $K,V$.

The equation at the bottom reinforces the main point: all three cases still use scaled dot-product attention. What changes is the mask $M$ and the provenance of the matrices. Once that is clear, the transition to training objectives becomes straightforward: each architecture defines which conditional distribution it is allowed to model, and the next step is to train those conditionals by maximum likelihood.

Having separated the Transformer into encoder, decoder, and cross-attention components, the next question is: what exactly do we optimize during training? The architecture gives us a conditional distribution over tokens, but training turns that distribution into a supervised learning problem repeated across every position in a sequence.

For an encoder-decoder Transformer, the target sequence $y_{1:m}$ is generated autoregressively conditioned on the source sequence $x_{1:n}$. That means we model the joint conditional probability by the chain rule:

$$p_\theta(y_{1:m}\mid x_{1:n}) = \prod_{t=1}^{m} p_\theta(y_t\mid y_{<t},x_{1:n})$$

This factorization is not an approximation by itself; it is just the probability chain rule. The modeling assumption enters through the Transformer parameterization of each conditional distribution $p_\theta(y_t\mid y_{<t},x_{1:n})$. At position $t$, the decoder is supposed to use the source representation from the encoder and the already generated target prefix $y_{<t}$, but not the future target tokens $y_{>t}$.

This is where teacher forcing enters. During training, we do not ask the model to generate its own prefix token by token and then learn from the resulting rollout. Instead, for every position $t$, we feed the decoder the ground-truth prefix. The model sees the correct previous tokens and is trained to predict the next one:

$$y_{<t} \longrightarrow y_t$$

The subtle but crucial detail is that the model may receive the full target sequence as a tensor during the forward pass, but the causal mask ensures that position $t$ can only attend to positions $<t$. Without this mask, the decoder could leak information from $y_t$ or $y_{>t}$, making the training loss artificially low and destroying the intended autoregressive semantics.

The maximum-likelihood objective asks us to maximize the probability of the observed target sequences under this factorization. Equivalently, we minimize the negative log-likelihood:

$$\mathcal{L}(\theta) = -\sum_{(x_{1:n},y_{1:m})\in\mathcal{D}} \sum_{t=1}^{m} \log p_\theta(y_t\mid y_{<t},x_{1:n})$$

In practice, this is exactly the usual cross-entropy loss over the vocabulary at every decoder position. The decoder outputs a vector of logits at each position; after a softmax, the probability assigned to the correct next token is selected, logged, negated, and summed. Because the correct prefix is supplied everywhere, every target position contributes a supervised training signal in one forward pass.

This is one of the key computational advantages of Transformer training. Although the probability model is autoregressive, the training computation is parallel over positions. We do not have to run the decoder once for $t=1$, then again for $t=2$, and so on. Instead, the causal mask gives each position the right information boundary, allowing all next-token predictions to be trained simultaneously:

• position $1$ predicts $y_1$ from the start context,
• position $2$ predicts $y_2$ from $y_1$,
• position $t$ predicts $y_t$ from $y_{<t}$,
• position $m$ predicts $y_m$ from $y_{<m}$.

For decoder-only language modeling, the same idea applies after removing the source sequence. A language model learns to predict each token from its left context:

$$\mathcal{L}(\theta) = -\sum_{x_{1:n}\in\mathcal{D}} \sum_{t=1}^{n} \log p_\theta(x_t\mid x_{<t})$$

This is the objective behind standard autoregressive pretraining. The “input” and “target” are shifted versions of the same sequence: the model consumes previous tokens and predicts the next token. Again, the causal mask is what makes it legitimate to process the entire sequence at once while preserving the left-to-right conditional structure.

There is an important distinction between training and decoding here. During training, teacher forcing conditions on the true prefix, so all positions are supervised in parallel. During inference, the true prefix is unavailable; the model must condition on tokens it has already generated. This creates exposure to its own mistakes, and it also makes decoding sequential in the output length. Thus, Transformers train highly parallelly but still generate autoregressively unless we change the modeling assumptions.

The visual summary below compresses this objective into three pieces: the autoregressive factorization, the encoder-decoder negative log-likelihood, and the decoder-only language modeling loss. The highlighted prefixes $y_{<t}$ and $x_{<t}$ are the conditioning contexts supplied under teacher forcing, while the causal mask marks the boundary that prevents a position from seeing future tokens.

The small token timeline reinforces the operational meaning of the equations: every position is trained as a next-token prediction problem, but the red barrier imposed by the causal mask keeps each prediction honest. This is the central training recipe for sequence Transformers: sum cross-entropy over examples and positions, with ground-truth prefixes and no future-token leakage.

Having defined the teacher-forced likelihood objective, we now need to be precise about what the model actually computes before that objective is evaluated. For an encoder-only Transformer, the forward pass is conceptually simple: convert a sequence of tokens into a sequence of contextual vectors, repeatedly allowing each position to retrieve information from other positions and then transform its own representation through a shared nonlinear map.

The input is a token sequence $x_{1:n}$. Since attention by itself is permutation-equivariant, the encoder must be given some representation of order. The usual first step is therefore to add a learned or fixed positional vector $p_i$ to each token embedding $e(x_i)$:

$$X \leftarrow [\,e(x_1)+p_1;\ldots;e(x_n)+p_n\,] \in \mathbb{R}^{n\times d_{\mathrm{model}}}.$$

Here each row of $X$ is the representation of one token position. The matrix shape matters: the encoder preserves the sequence length $n$ throughout the stack, while updating the $d_{\mathrm{model}}$-dimensional representation at each position. This is one reason encoders are reusable: their output is still a sequence, not a single collapsed vector.

Each encoder layer then applies two sublayers. The first is multi-head self-attention, where every position forms queries, keys, and values from the current sequence representation. Within each head, attention weights are computed as

$$A=\operatorname{softmax}\!\left(\frac{QK^\top}{\sqrt{d_k}}+M\right).$$

The mask $M$ is especially important in batched training. For a standard bidirectional encoder, we usually do not use a causal mask, because each token is allowed to attend to tokens on both its left and right. But if examples have been padded to a common length, padded positions must not participate as real content. This is handled by adding a padding mask before the row-wise softmax, typically using large negative values so that attention probability on padded keys becomes essentially zero.

After attention, the encoder does not simply replace $X$ with $\operatorname{MHA}(X)$. Instead it uses a residual connection, dropout, and layer normalization:

$$Z \leftarrow \operatorname{LN}\bigl(X + \operatorname{Dropout}(\operatorname{MHA}(X))\bigr).$$

The residual path is not just an implementation detail. It gives the layer an easy way to preserve existing information, improves gradient flow through deep stacks, and lets attention learn corrections rather than recomputing the entire representation from scratch. Layer normalization then stabilizes the scale of activations at each position, which becomes increasingly important as $L$, the number of layers, grows.

The second sublayer is the position-wise feed-forward network:

$$X \leftarrow \operatorname{LN}\bigl(Z + \operatorname{Dropout}(\operatorname{FFN}(Z))\bigr).$$

“Position-wise” means the same multilayer perceptron is applied independently to each row of $Z$. Attention is the mechanism that mixes information across positions; the feed-forward network is the mechanism that applies a richer nonlinear transformation within each position. This separation is one of the clean design principles of the Transformer block:

• Self-attention: communicates across tokens.
• FFN: transforms each token representation independently.
• Residual connections: preserve information and ease optimization.
• Layer normalization: keeps activations numerically stable.
• Dropout: regularizes both sublayers during training.

A subtle but useful way to view the encoder is as a repeated refinement process. At layer $1$, a token’s representation may mostly encode lexical identity and position. After several layers, that same row can encode syntactic role, semantic relationships, discourse context, or task-relevant features—while still occupying the same sequence slot. The algorithm returns the final matrix $X$, whose rows are contextual embeddings. Depending on the task, this matrix might feed a classifier, a retrieval head, a token-level predictor, or the cross-attention module of a decoder.

The main failure mode to watch for is confusing the encoder mask with the decoder mask. In an encoder, the mask is usually about padding, not causality. If we accidentally apply a causal mask inside a bidirectional encoder, we restrict the model unnecessarily and change its equivariance properties. Conversely, if we forget the padding mask, real tokens may attend to padding embeddings, allowing meaningless positions to contaminate the contextual representations.

The visual below packages this forward pass as an algorithm: initialize token-plus-position representations, loop through $L$ identical encoder blocks, apply masked multi-head self-attention with residual normalization, then apply the shared position-wise feed-forward update with another residual normalization. The right-hand stack view is a useful mental model: the sequence enters at the bottom, is lifted into $X$, passes through repeated encoder layers, and exits as a contextual sequence representation.

Read the pseudocode not as a low-level implementation prescription, but as the mathematical skeleton of the encoder. Actual implementations may choose pre-norm instead of post-norm, fuse projections for efficiency, or batch many sequences together, but the invariant structure remains the same: self-attention mixes tokens, the FFN transforms positions, and the stack preserves sequence shape while enriching representation quality.