LoRA Fine-Tuning: Low-Rank Adaptation of Large Neural Networks

Before we get to LoRA itself, it is worth slowing down on the pressure point that made methods like LoRA necessary. Modern pretrained models are powerful precisely because they concentrate enormous general-purpose knowledge into a parameter vector $\theta_0$. But that same scale turns the most obvious adaptation strategy—just fine-tune everything—into a systems problem as much as a learning problem.

Suppose we start with a pretrained model $f_{\theta_0}$ and a supervised downstream dataset

$$\mathcal{D}=\{(x_i,y_i)\}_{i=1}^{n}.$$

For a task such as sentiment classification, medical extraction, instruction following, or customer-specific generation, adaptation means changing something about the model so that its predictions $\hat{y}_i$ better match the desired outputs $y_i$. Abstractly, we choose some trainable parameters $\phi$ and minimize the empirical loss

$$\mathcal{L}(\phi) = \frac{1}{n}\sum_{i=1}^{n}\ell(\hat{y}_i,y_i), \qquad \hat{y}_i=f_{\theta}(x_i).$$

The important detail is that $\phi$ does not have to be the entire model. It simply denotes whatever parameters we decide are allowed to move during adaptation. In full fine-tuning, we make the maximal choice: initialize $\theta=\theta_0$, set $\phi=\theta$, and optimize every parameter of the pretrained model.

This is a very strong baseline. Full fine-tuning gives the optimizer direct access to every layer, every attention projection, every MLP weight, every embedding table, and every normalization parameter. If the downstream task needs a subtle redistribution of features throughout the network, full fine-tuning can in principle make those changes. That flexibility is one reason it often performs well when we have enough data, compute, and careful regularization.

The bottleneck is that flexibility is expensive. A model with billions of parameters is not just expensive to pretrain; it is expensive to repeatedly copy, optimize, store, serve, version, and audit. During training, full fine-tuning typically requires optimizer state for every trainable parameter. With Adam-like optimizers, that can mean storing the parameter, its gradient, and multiple moment estimates. The memory footprint can become several times larger than the model weights alone.

The deployment cost is even more direct. If we fine-tune a separate full model for each task or customer, then each adaptation produces another full-sized parameter vector:

$$\theta^{(1)},\theta^{(2)},\ldots,\theta^{(T)}.$$

Even if each $\theta^{(t)}$ differs only slightly from the original $\theta_0$, naive full fine-tuning stores every adapted copy independently. For a 7B, 13B, or 70B parameter model, this quickly becomes impractical. The problem is not merely training one model once; the problem is supporting many adapted models.

There is also a statistical subtlety. Downstream datasets are often much smaller than the pretraining corpus. When $n$ is small and $\theta$ has billions of trainable degrees of freedom, the model has enough capacity to memorize idiosyncrasies of the task dataset. Good fine-tuning practice therefore relies on learning-rate schedules, regularization, early stopping, validation sets, and sometimes freezing parts of the network. Full fine-tuning is powerful, but it is not automatically data-efficient or operationally convenient.

This motivates the central question behind parameter-efficient fine-tuning:

Can we keep the pretrained weights $\theta_0$ frozen, while letting a much smaller set of parameters $\phi$ carry the task-specific adaptation?

If the answer is yes, then we can reuse one large shared model and store only a small task-specific delta for each downstream use case. Instead of maintaining many full copies of $\theta$, we maintain one base model plus many compact adaptations. LoRA will instantiate this idea by restricting the update to certain weight matrices and forcing that update to have low rank, but the motivation starts here: most of the pretrained model should remain a reusable shared asset.

A useful way to frame the trade-off is:

• Full fine-tuning: maximum flexibility, but every task may require a full model copy.
• Frozen base model: maximum reuse, but no adaptation unless we add trainable task-specific parameters.
• Parameter-efficient adaptation: preserve most of the pretrained model while learning a small, targeted modification.

The visual below compresses this argument into a left-to-right bottleneck: a small task dataset feeds into a huge pretrained model, full fine-tuning produces separate large adapted models, and the cost grows with the number of tasks or customers. The key asymmetry is that the task data and desired behavioral change may be small, while the object being duplicated is enormous.

The small $\phi$ placeholder in the diagram points toward the idea LoRA will develop next. Instead of asking every parameter in $\theta_0$ to move, we will ask whether adaptation can live in a much lower-dimensional space—small enough to store and train cheaply, but expressive enough to recover much of the performance of full fine-tuning.

The optimization view from the previous section makes full fine-tuning look deceptively simple: start from a pretrained parameter vector $\theta_0$, optimize the downstream loss, and obtain an adapted parameter vector $\theta$. Conceptually, that is clean. Operationally, however, it hides the main scaling problem: after fine-tuning, $\theta$ is no longer a universal shared object. It is now task-specific.

If we adapt the same pretrained model to one downstream task, this may be acceptable. But the whole point of large pretrained models is that they are reused across many tasks: summarization, code generation, retrieval-augmented QA, domain-specific chat, instruction following, classification, extraction, and so on. With full fine-tuning, adapting one base model to 100 downstream tasks gives 100 distinct full model checkpoints. Each checkpoint contains a complete copy of the model parameters, even if the actual task-specific change from $\theta_0$ is small.

For a single dense weight matrix $W_0 \in \mathbb{R}^{d \times k}$, full fine-tuning treats every entry as trainable. The number of trainable parameters in that matrix is therefore

$$p_{\mathrm{full}} = dk.$$

This looks harmless at the matrix level, but modern transformers contain many such matrices: attention projections, MLP projections, embeddings, output heads, and normalization-related parameters. Full fine-tuning scales with the size of the entire model, not with the amount of information needed to specialize the model to the task.

The training memory cost is even worse than the checkpoint size suggests. During training, we do not only store the parameters themselves. We also store gradients, activations needed for backpropagation, and optimizer state. For Adam-like optimizers, each trainable parameter typically carries additional moment estimates, often the first and second moments. Abstracting away implementation details, the memory footprint associated with trainable parameters can be summarized as

$$M_{\mathrm{train}} \approx (c_{\mathrm{param}} + c_{\mathrm{opt}})p_{\mathrm{full}},$$

where $c_{\mathrm{param}}$ accounts for storing trainable weights and related parameter-level quantities, while $c_{\mathrm{opt}}$ accounts for optimizer state. The important point is not the exact constant; it is the scaling. If every parameter is trainable, Adam allocates optimizer state for every parameter.

This creates a practical failure mode that is easy to underestimate. The base model may be shared in theory, but full fine-tuning destroys that sharing in deployment. Each task now needs its own full checkpoint. Serving many variants requires model routing, checkpoint loading, memory residency decisions, version management, and duplicated storage. If the base model has billions of parameters, even “just one more fine-tuned task” can become expensive.

The key inefficiency is that full fine-tuning assumes the task-specific solution must be represented as an entirely new point $\theta$ in the full parameter space. But empirically, many downstream adaptations appear to require only a comparatively small change in behavior. We would like to capture that change without rewriting and storing the entire model. In other words, the desired pattern is:

• keep the pretrained parameters $\theta_0$ shared and frozen;
• learn a much smaller task-specific object $\phi$;
• combine $\theta_0$ and $\phi$ at inference time to recover task-adapted behavior.

This is the motivation behind parameter-efficient fine-tuning. Rather than asking, “How do we find a new full model for each task?”, we ask, “Can we represent the task-specific update compactly?” LoRA will answer this by constraining updates to certain weight matrices to have low rank, but before getting there, it is important to see the failure case clearly: full fine-tuning pays the full model cost even when the useful adaptation may be much smaller.

The visual below compresses this contrast into two patterns. On the left is the full fine-tuning regime: one shared pretrained model gives rise to many separate full checkpoints, with storage and Adam state growing with the full parameter count. On the right is the parameter-efficient goal: keep $f_{\theta_0}$ frozen and shared, while attaching small task-specific parameters $\phi$.

The equation at the bottom anchors the intuition mathematically. For a fully trainable matrix, $p_{\mathrm{full}}=dk$, and the training footprint scales like $(c_{\mathrm{param}}+c_{\mathrm{opt}})p_{\mathrm{full}}$. The rest of the lecture is about replacing that scaling law with something much smaller, while preserving most of the quality benefits of fine-tuning.

The “one full copy per task” failure mode suggests a natural repair: stop treating every downstream task as a reason to duplicate the whole model. If the pretrained parameters $\theta_0$ already encode broadly useful linguistic, visual, or multimodal structure, then perhaps task adaptation should be a small correction rather than a complete rewrite. This is the core motivation behind parameter-efficient fine-tuning: keep the base model frozen, and learn only a comparatively tiny set of task-specific parameters $\phi$.

In abstract form, most PEFT methods follow the same pattern:

$$\theta_0 \text{ frozen}, \qquad \phi \text{ trainable}, \qquad |\phi| \ll |\theta_0|.$$

This changes the economics of fine-tuning. Instead of storing and serving a separate full model for each task, we store one shared pretrained backbone plus many small task modules. Training also becomes cheaper because gradients, optimizer states, and parameter updates are needed only for $\phi$, not for the full $\theta_0$. For very large models, this distinction is not cosmetic: optimizer states alone can multiply memory requirements several times over the raw parameter count.

But “freeze the model and train something small” is not a complete algorithm. The hard question is where to put the small trainable object so that it has enough influence over the computation. Different PEFT methods answer this question differently, and their trade-offs are mostly about the location and form of $\phi$.

Prompt tuning puts the trainable parameters at the input side. Instead of updating the transformer weights, we prepend learned continuous prompt vectors to the input embeddings. This can be extremely parameter-efficient because the trainable object is just a small collection of vectors. However, it also means the model is adapted indirectly: the only way the task signal can affect the network is by changing the effective input sequence. That can be elegant, but it may be too weak for tasks requiring deeper internal changes, and it consumes part of the model’s context budget.

Prefix tuning is more targeted. Rather than adding learned vectors only at the input embedding layer, it learns prefix states that are injected into attention, often as additional key/value vectors. This gives the task parameters a more direct handle on attention patterns: the model can attend to learned task-specific memory at each layer. The cost is that these prefixes increase the effective attention context length, which can add computation and memory overhead during inference. The base weights remain frozen, but the runtime attention path is now larger.

Adapters take a different route: they insert small neural modules inside the network, commonly bottleneck MLPs placed between or within transformer sublayers. A typical adapter maps a hidden state down to a low-dimensional space, applies a nonlinearity, maps it back up, and adds the result residually. This is more expressive than input-side prompt methods because it creates new task-specific computation throughout the model. The downside is architectural surgery. The model’s forward graph now contains extra modules, which can introduce latency and complicate deployment, especially when we care about highly optimized inference kernels.

LoRA belongs to this PEFT family, but its design choice is more surgical: instead of changing the input sequence, extending attention context, or inserting new nonlinear modules, it modifies the effective weights of existing linear layers. The pretrained weight $W_0$ is frozen, but the layer behaves as though it had received an additive update:

$$z = (W_0 + \Delta W_{\mathrm{LoRA}})h + b_0.$$

The crucial constraint is that this update is not a full dense matrix. LoRA parameterizes it as a low-rank product:

$$\Delta W_{\mathrm{LoRA}} = sBA,$$

where $A$ and $B$ are trainable low-rank factors and $s$ is a scaling factor. If $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$, then a full update would require $d_{\text{out}}d_{\text{in}}$ trainable parameters. LoRA instead uses something like

$$A \in \mathbb{R}^{r \times d_{\text{in}}}, \qquad B \in \mathbb{R}^{d_{\text{out}} \times r},$$

for a total of

$$r(d_{\text{in}} + d_{\text{out}})$$

trainable parameters. When $r \ll \min(d_{\text{in}}, d_{\text{out}})$, this is dramatically smaller than a dense update.

This design gives LoRA an important deployment advantage over many earlier PEFT methods: after training, the learned update can often be merged into the frozen weight by replacing $W_0$ with $W_0 + \Delta W_{\mathrm{LoRA}}$. At inference time, the layer can remain an ordinary linear transformation. There is no extra prompt length, no additional attention prefix, and no new adapter branch that must be executed separately. The trade-off is that LoRA’s expressiveness is controlled by the rank $r$. If $r$ is too small, the update may be unable to represent the adaptation the task needs; if $r$ is too large, we lose some of the parameter-efficiency and memory benefits.

So the PEFT landscape can be summarized by a simple comparison:

• Prompt tuning: smallest and simplest, but influences the model indirectly through inputs.
• Prefix tuning: more direct control over attention, but increases attention length.
• Adapters: expressive internal computation, but add modules and inference overhead.
• LoRA: trainable updates inside existing linear weights, with a mergeable low-rank form.

The visual below consolidates this comparison. The first three methods all preserve the frozen pretrained backbone, but they attach task-specific capacity in different places: before the model, inside attention state, or as inserted modules. LoRA is highlighted because it keeps the computation closest to the original linear layer while still allowing a learned task-specific correction.

The equations underneath the comparison are the key transition into the next idea. LoRA is not merely “another small module”; it is a hypothesis about the geometry of fine-tuning itself: perhaps the useful update $\Delta W$ does not need to occupy the full matrix space. If task adaptation lives in a low-dimensional subspace, then the factorization $\Delta W_{\mathrm{LoRA}} = sBA$ is not just a memory trick — it is a structured constraint on how the pretrained model is allowed to move.

The tension with earlier PEFT methods is that they save parameters by adding something around the pretrained model: prompts, adapters, extra bottlenecks, or task-specific modules. That can work well, but it raises an important question: do we really need to alter the architecture or insert new computation paths, or can we express task adaptation as a small change to the weights that are already there?

LoRA starts from a very direct view of fine-tuning. Suppose a pretrained linear layer has weight matrix $W_0 \in \mathbb{R}^{d \times k}$. In ordinary full fine-tuning, we replace it with a new learned matrix $W$. Equivalently, we can write the final weight as the pretrained matrix plus a task-specific displacement:

$$W = W_0 + \Delta W, \qquad W_0 \in \mathbb{R}^{d \times k}\ \text{frozen}.$$

This decomposition is conceptually useful because it separates two roles. The pretrained matrix $W_0$ already contains broad linguistic, visual, or multimodal structure learned from large-scale pretraining. The downstream task does not usually require relearning all of that structure from scratch. Instead, it may only need to steer the pretrained transformation in a relatively small number of directions.

The central LoRA hypothesis is therefore not that the pretrained weight itself is simple. $W_0$ may be full-rank, dense, and highly expressive. The claim is about the update induced by downstream fine-tuning:

$$\operatorname{rank}(\Delta W) \ll \min(d,k).$$

In words: although $\Delta W$ has the same shape as $W_0$, the meaningful change needed for a task may lie in a much lower-dimensional subspace. This is plausible in large pretrained models because many downstream tasks are not asking the model to invent entirely new representations; they are asking it to recombine, emphasize, suppress, or redirect features that already exist.

A useful geometric picture is to think of full fine-tuning as allowing movement in every direction in the $d \times k$-dimensional weight space. LoRA assumes that the useful displacement from $W_0$ lives near a much smaller manifold or subspace. Rather than learning every entry of $\Delta W$ independently, we restrict $\Delta W$ to have low rank by factorizing it:

$$\Delta W_{\mathrm{LoRA}} = sBA, \qquad A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad s=\alpha/r.$$

Here $r$ is the LoRA rank, usually chosen much smaller than $d$ or $k$. The matrix $A$ projects the input into an $r$-dimensional adaptation space, and $B$ maps that low-dimensional signal back into the output dimension. The scalar $s=\alpha/r$ controls the scale of the update, making the adaptation strength easier to tune across different ranks.

This factorization immediately gives the rank constraint. Since $BA$ is the product of a $d \times r$ matrix and an $r \times k$ matrix,

$$\operatorname{rank}(BA) \leq \min(\operatorname{rank}(B), \operatorname{rank}(A)) \leq r.$$

So LoRA does not merely hope that the update will be low-rank; it enforces that constraint by construction. The learned update cannot use more than $r$ independent directions, no matter how large the original matrix is.

The parameter savings follow from the same structure. A full update $\Delta W$ would require $dk$ trainable parameters. LoRA learns only $A$ and $B$, requiring

$$rk + dr = r(d+k)$$

parameters. When $r \ll \min(d,k)$, this is dramatically smaller than $dk$. For example, if $d=k=4096$ and $r=8$, a full update has over 16 million parameters, while the LoRA update has only $8(4096+4096)=65{,}536$ parameters for that layer.

The subtle but important point is that LoRA turns fine-tuning into subspace learning around a strong pretrained solution. We are not training a small model instead of a large one. We are using the large model as a fixed base and learning a compact task-specific displacement. This is why LoRA can often preserve the capabilities of the pretrained model while adapting efficiently: the expressive burden remains mostly in $W_0$, and the trainable part only needs to encode the difference.

There are also failure modes hidden inside the hypothesis. If a task genuinely requires many independent changes to a layer, a very small rank $r$ may underfit. If the inserted LoRA modules are placed in layers or projections that are not important for the task, the low-rank update may be parameter-efficient but ineffective. And if the downstream distribution is far from pretraining, the assumption that adaptation is a small low-dimensional displacement may become weaker. LoRA is powerful precisely when the pretrained representation is already close and the task mostly needs targeted steering.

The visual below can be read as a compact summary of this idea. The large frozen matrix $W_0$ represents the stable pretrained transformation. The small low-rank branch, formed by $A$ and $B$, represents the lightweight trainable update $\Delta W_{\mathrm{LoRA}} = sBA$. The adapted weight is not a completely new matrix learned from scratch; it is the frozen base plus this constrained task-specific correction.

The key contrast is visual as much as mathematical: $W_0$ remains large and expressive, while the learned update is deliberately narrow. The annotation $r \ll \min(d,k)$ captures the LoRA bet: most of the useful downstream movement can be expressed through a small number of adaptation directions.

Having argued that task-specific changes may live in a much smaller subspace than the full parameter space, we now zoom in to the smallest object where LoRA actually acts: one linear layer. This is the right level of abstraction because transformer blocks are built from many affine maps—query, key, value, output projections, MLP projections—and fine-tuning ultimately changes the matrices inside those maps.

Consider a pretrained linear layer receiving an activation vector $h \in \mathbb{R}^{k}$ and producing an output vector $z \in \mathbb{R}^{d}$. Before any adaptation, the layer computes

$$z = W_0 h + b_0,$$

where $W_0 \in \mathbb{R}^{d \times k}$ is the pretrained weight matrix and $b_0 \in \mathbb{R}^{d}$ is the pretrained bias. The subscript $0$ is important: it reminds us that these parameters come from pretraining and are treated as the reference point around which adaptation happens.

Full fine-tuning would simply allow $W_0$ and $b_0$ to move freely during task training. But LoRA begins from a different framing. Instead of saying “we learn a new weight matrix,” we say: the adapted model uses an effective weight

$$W = W_0 + \Delta W, \qquad \Delta W \in \mathbb{R}^{d \times k}.$$

Here, $\Delta W$ is the task-specific correction to the pretrained matrix. If $\Delta W = 0$, we recover the original model exactly. If $\Delta W$ is unrestricted, this is just another way of writing ordinary weight fine-tuning. The key move in LoRA will be to keep $W_0$ fixed and learn only a structured version of $\Delta W$.

This additive view is more than notation. It separates two roles that are entangled in full fine-tuning:

• $W_0$ stores broad pretrained knowledge accumulated from large-scale training.
• $\Delta W$ stores the task-specific displacement needed for adaptation.

That distinction is what makes parameter-efficient fine-tuning possible. We do not need to rewrite the pretrained model from scratch; we need to learn a correction that steers its existing computation toward a new task or distribution.

There is also a subtle but useful assumption here: LoRA treats adaptation as a perturbation around a strong pretrained solution. That assumption is usually reasonable when the downstream task is related to the model’s pretraining distribution, but it can fail when the new task demands behavior far outside the pretrained model’s capabilities. In such cases, a small or low-rank $\Delta W$ may not have enough expressive power, and full fine-tuning—or a larger adaptation mechanism—may be necessary.

At this point, $\Delta W$ is still a full $d \times k$ matrix. So we have not yet saved parameters. We have only changed our perspective from “learn the whole weight” to “learn an update to a frozen weight.” The next step will be to constrain $\Delta W$ to have low rank, but the additive decomposition itself is the core abstraction:

$$\text{adapted computation} \;=\; (W_0 + \Delta W)h + b_0.$$

In LoRA, $W_0$ and typically $b_0$ remain frozen during training. Gradients flow through the layer as usual, but only the parameters used to construct $\Delta W$ are updated. This preserves the pretrained matrix exactly, reduces optimizer state, and allows multiple task-specific adaptations to be stored separately while sharing the same base model.

The same abstraction applies directly to transformer projection matrices such as $W_Q$, $W_K$, $W_V$, and $W_O$. Each is just a linear map from one activation space to another. LoRA does not require a special transformer-specific derivation at this stage; it only requires identifying which linear maps should receive learned additive updates.

The visual below condenses this idea into one frozen affine layer and its adapted counterpart. The blue components represent the pretrained computation $z = W_0h + b_0$, while the orange update $\Delta W$ marks the only part that will later become trainable under LoRA’s low-rank parameterization.

Read it as a bridge between standard fine-tuning and LoRA: first define the effective adapted weight $W = W_0 + \Delta W$; then, in the next step, restrict the shape of $\Delta W$ so that adaptation becomes dramatically cheaper without discarding the pretrained layer.

Now that we have isolated a single frozen linear layer, the next step is to ask what full fine-tuning actually changes at that layer. The usual description says that fine-tuning “updates the weights,” but for LoRA the more useful viewpoint is slightly different: full fine-tuning learns an additive correction to the pretrained matrix.

Start from the frozen layer

$$z = W_0 h + b_0,$$

where $W_0 \in \mathbb{R}^{d \times k}$, $h \in \mathbb{R}^k$, and $z \in \mathbb{R}^d$. The matrix $W_0$ is the pretrained weight we inherited from the base model. During full fine-tuning, we replace it by a new adapted matrix $W$. But any such adapted matrix can always be written as

$$W = W_0 + \Delta W, \qquad \Delta W \in \mathbb{R}^{d \times k}.$$

This is not yet an approximation or a LoRA assumption. It is just algebra. Given any final fine-tuned weight $W$, the difference

$$\Delta W = W - W_0$$

is the update that full fine-tuning has learned relative to the pretrained model.

Substituting this decomposition back into the layer gives

$$z = (W_0 + \Delta W)h + b_0.$$

By distributing over $h$, we get

$$z = W_0h + \Delta W h + b_0.$$

This expression is the key abstraction. The adapted layer can be understood as the original pretrained computation $W_0h + b_0$, plus an additional task-specific correction $\Delta W h$. In other words, fine-tuning does not need to be viewed as replacing the pretrained model wholesale. At the level of one linear layer, it is equivalent to keeping the pretrained transformation and adding a learned residual transformation.

Under full fine-tuning, however, the update $\Delta W$ is completely unconstrained. Since $\Delta W$ has the same shape as $W_0$, it contains

$$p_{\mathrm{full}} = dk$$

free parameters for this layer alone. Every entry of the $d \times k$ matrix may move independently. If the layer maps a $k$-dimensional activation into a $d$-dimensional output, full fine-tuning allows an arbitrary linear correction from $\mathbb{R}^k$ to $\mathbb{R}^d$.

That flexibility is powerful, but it is also expensive. In modern transformers, the major weight matrices are large, and there are many of them. Updating all of their entries means we must:

• store optimizer states for every trainable weight,
• backpropagate gradients through every adapted parameter,
• maintain a separate full-size copy of the model for each fine-tuned task,
• and risk overfitting when the downstream dataset is small relative to the number of trainable parameters.

The subtle point is that LoRA does not begin by removing the additive correction $\Delta W h$. It keeps exactly this residual-update interpretation. What LoRA changes is the space of matrices that $\Delta W$ is allowed to occupy. Full fine-tuning says: “learn any matrix $\Delta W \in \mathbb{R}^{d \times k}$.” LoRA will say: “learn a structured, low-rank matrix $\Delta W$ that is much cheaper to represent.”

This distinction matters because it explains why LoRA can be inserted into an existing pretrained model without changing the basic forward computation. The layer still produces the frozen pretrained contribution $W_0h$, and it still adds an adaptation term. The efficiency gain comes from parameterizing that adaptation term more carefully, not from inventing a new kind of layer.

The visual below compresses this logic into three pieces: the original frozen layer, the decomposition $W = W_0 + \Delta W$, and the adapted computation where the correction appears explicitly as $\Delta W h$. The color separation is important: pretrained quantities remain frozen, while the update matrix is the trainable object.

It also highlights the cost of doing this without constraints. The orange $d \times k$ block represents an unconstrained update matrix with $dk$ free parameters. This is the baseline LoRA will improve upon: not by denying that full fine-tuning learns an additive update, but by restricting that update to a much smaller low-rank family.

Having written full fine-tuning as a frozen pretrained map plus an additive correction, the natural next question is: how expressive does that correction really need to be? If a layer originally computes

$$y = W_0 h + b_0,$$

then full fine-tuning replaces $W_0$ by $W_0 + \Delta W$, so the adapted layer becomes

$$y = W_0 h + \Delta W h + b_0.$$

The key observation is that, for this layer, all task-specific adaptation enters through the vector $\Delta W h$. We do not directly care about $\Delta W$ as an abstract matrix; we care about the directions in activation space that it can add to the frozen pretrained computation.

A completely unrestricted $\Delta W \in \mathbb{R}^{d \times k}$ can represent any linear correction from the input activation space $\mathbb{R}^k$ to the output activation space $\mathbb{R}^d$. Geometrically, this means the update may use as many as $\min(d,k)$ independent output directions. That is powerful, but expensive: it requires learning $dk$ new degrees of freedom for a single weight matrix. In large transformers, many such matrices are enormous, so repeating this across attention and MLP layers quickly becomes the dominant storage and optimization cost.

LoRA’s central bet is that useful downstream adaptation often lives in a much smaller subspace. Instead of allowing $\Delta W$ to be arbitrary, we constrain it to have low rank:

$$\operatorname{rank}(\Delta W) \le r, \qquad r \ll \min(d,k).$$

This is not merely a parameter-count trick. It is a geometric restriction on what kinds of changes the model can make. A rank-$r$ matrix can only map inputs into an output subspace of dimension at most $r$. So although $\Delta W h$ is still a vector in $\mathbb{R}^d$, as $h$ varies it can only move within at most $r$ learned directions inside that $d$-dimensional output space.

The most useful way to see this is through factorization. Any rank-at-most-$r$ update can be represented as the product of two thinner matrices:

$$\Delta W = BA,$$

where

$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}.$$

Then the update applied to an activation $h \in \mathbb{R}^k$ becomes

$$\Delta W h = B(Ah).$$

This equation is the geometric heart of LoRA. The matrix $A$ first compresses the original $k$-dimensional activation into $r$ coordinates:

$$h \in \mathbb{R}^k \quad \longmapsto \quad Ah \in \mathbb{R}^r.$$

Then $B$ expands those $r$ coordinates back into a $d$-dimensional update vector:

$$Ah \in \mathbb{R}^r \quad \longmapsto \quad B(Ah) \in \mathbb{R}^d.$$

So the update is forced through an $r$-dimensional bottleneck. The model can still produce a full-sized correction vector, but that correction must be assembled from at most $r$ learned basis directions: the columns of $B$. The compressed vector $Ah$ supplies the coefficients for combining those directions.

This also clarifies both the strength and the limitation of LoRA. If the task-specific change really is concentrated in a low-dimensional subspace, then a small $r$ can capture it efficiently. But if adaptation requires many independent directions — for example, if the downstream task demands a broad reorganization of the layer’s representation — then too small a rank may underfit. In that case, the bottleneck is not just saving parameters; it is actively preventing certain updates from being represented.

LoRA usually adds one more scaling factor:

$$\Delta W_{\mathrm{LoRA}} = sBA, \qquad s = \alpha/r.$$

The scalar $s$ controls the magnitude of the low-rank branch relative to the frozen pretrained path. This matters because the matrices $A$ and $B$ are trained from initialization, while $W_0$ already contains a large amount of pretrained structure. The scaling helps make the injected update numerically stable and comparable across different choices of rank $r$.

A compact way to remember the whole construction is:

• Full fine-tuning: learn an arbitrary $\Delta W \in \mathbb{R}^{d \times k}$.
• Low-rank adaptation: learn $\Delta W = BA$, with a bottleneck dimension $r$.
• Geometric effect: updates live in at most $r$ learned output directions.
• LoRA form: use the scaled update $\Delta W_{\mathrm{LoRA}} = sBA$.

The visual below consolidates this idea as a flow: the input activation $h$ is first passed through $A$, producing a narrow $r$-dimensional bottleneck $Ah$, and then through $B$, producing the full output update $\Delta W h$. The frozen pretrained computation remains in the background, while the colored low-rank branch shows exactly where adaptation is being inserted.

The important thing to notice is not just that the middle vector is smaller. The bottleneck is what enforces the rank constraint. Since every update must pass through $r$ coordinates before being expanded, the final correction can only span an $r$-dimensional family of output movements. That is the geometric meaning of “low rank” in LoRA: compress to a small learned coordinate system, then expand into a controlled task-specific update.

The geometric picture of low rank gives us the right intuition: a low-rank matrix can only move inputs through a small-dimensional subspace before expanding them back out. LoRA turns that intuition into a concrete parameterization for fine-tuning. Instead of learning an arbitrary dense update $\Delta W \in \mathbb{R}^{d \times k}$, we force the update to pass through an $r$-dimensional bottleneck.

For a pretrained weight matrix $W_0 \in \mathbb{R}^{d \times k}$, full fine-tuning would replace it by

$$W = W_0 + \Delta W,$$

where $\Delta W$ has the same shape as $W_0$ and contains $dk$ trainable parameters. LoRA keeps $W_0$ frozen and constrains the trainable update to have the factorized form

$$\Delta W_{\mathrm{LoRA}} = sBA, \qquad s = \alpha/r .$$

Here

$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r}, \qquad r \ll \min(d,k).$$

The multiplication order is important. An input vector in $\mathbb{R}^k$ is first projected down by $A$ into $\mathbb{R}^r$, then projected back up by $B$ into $\mathbb{R}^d$. The scalar $s=\alpha/r$ controls the magnitude of the LoRA update; it does not change the rank or the number of trainable parameters. It is best understood as a scaling convention that helps make different choices of $r$ comparable during optimization.

The key theorem is simple but powerful:

$$\operatorname{rank}(\Delta W_{\mathrm{LoRA}}) \le r .$$

This is the formal version of the bottleneck story. No matter how large $d$ and $k$ are, the product $BA$ cannot have rank larger than the intermediate dimension $r$. The update may live inside a huge $d \times k$ matrix, but its degrees of freedom are constrained to flow through an $r$-dimensional channel.

That constraint is exactly what makes LoRA parameter-efficient. Instead of training all $dk$ entries of $\Delta W$, we train only the entries of $A$ and $B$:

$$p_{\mathrm{LoRA}} = rk + dr = r(k+d).$$

By contrast, a full update would require

$$p_{\mathrm{full}} = dk.$$

So the trainable-parameter fraction is

$$\rho = \frac{p_{\mathrm{LoRA}}}{p_{\mathrm{full}}} = \frac{r(k+d)}{dk}.$$

When $r \ll \min(d,k)$, this ratio can be very small. For example, if $d=k=4096$ and $r=8$, then full fine-tuning uses over 16 million trainable parameters for that matrix, while LoRA uses only $8(4096+4096)=65{,}536$. That is about $0.39\%$ of the full parameter count for the layer.

There is a subtle but important distinction here: LoRA is not saying the weight matrix itself is low rank. The pretrained matrix $W_0$ remains full-size and may be full-rank. LoRA only constrains the change applied during adaptation:

$$W = W_0 + \Delta W_{\mathrm{LoRA}}.$$

This matters because the model can still use the rich representation already stored in $W_0$. LoRA merely assumes that the task-specific correction needed for fine-tuning lies in a much smaller subspace than the original model weights.

This assumption can fail. If a new task requires many independent directions of change in a layer, then a very small $r$ may underfit. Increasing $r$ gives the update more expressive capacity, but also increases memory, optimizer state, and potential overfitting. The practical appeal of LoRA comes from the empirical observation that many adaptation tasks do not require a full-rank update everywhere; a modest bottleneck often captures most of the useful task-specific movement.

The visual below compresses the theorem into one picture: an unconstrained dense update is replaced by two thinner matrices, $A$ and $B$, with a narrow $r$-dimensional passage between them. The green bottleneck is the reason for the rank bound, while the separate parameter counts remind us why the method is attractive computationally.

It is useful to read the diagram from left to right as a computation and from top to bottom as a theorem. Computationally, inputs pass through $A$, then $B$, then the scalar $s$. Mathematically, the same factorization guarantees $\operatorname{rank}(\Delta W_{\mathrm{LoRA}})\le r$ and reduces the number of trainable parameters from $dk$ to $r(k+d)$.

The theorem gives us the punchline; now we should make the proof feel almost inevitable. LoRA’s efficiency is not a mysterious property of transformers or attention layers. It comes from a simple linear-algebra fact: if we force an update matrix to factor through a small $r$-dimensional bottleneck, then the update cannot have rank larger than $r$, and the number of trainable parameters is the size of the two skinny factors rather than the size of the full matrix.

Suppose a pretrained layer contains a frozen weight matrix $W_0 \in \mathbb{R}^{d \times k}$. Full fine-tuning would learn an arbitrary update $\Delta W \in \mathbb{R}^{d \times k}$, so the adapted weight would be

$$W = W_0 + \Delta W.$$

LoRA restricts the update to the factorized form

$$\Delta W_{\mathrm{LoRA}} = sBA, \qquad A \in \mathbb{R}^{r \times k}, \quad B \in \mathbb{R}^{d \times r}.$$

Here $r$ is the LoRA rank, usually chosen much smaller than both $d$ and $k$, and $s$ is a scalar scaling factor. The important structural point is that the update does not map directly from $\mathbb{R}^k$ to $\mathbb{R}^d$ with a fully flexible $d \times k$ matrix. Instead, it first passes through an $r$-dimensional intermediate space via $A$, and then maps back up to $d$ dimensions via $B$.

That bottleneck immediately limits the rank. One way to see this is column-wise. The product $BA$ has $k$ columns, and each column is obtained by multiplying $B$ by the corresponding column of $A$. Therefore every column of $BA$ lies in the column space of $B$. Since $B$ has only $r$ columns, its column space has dimension at most $r$. Thus

$$\operatorname{rank}(BA) \le \operatorname{rank}(B) \le r.$$

Equivalently, we could reason from the other side: $A$ has only $r$ rows, so $\operatorname{rank}(A) \le r$, and the rank of a product cannot exceed the rank of either factor. Both perspectives say the same thing: the factorization forces the update to move within a low-dimensional subspace of possible full updates.

The scalar $s$ does not change this conclusion. If $s \neq 0$, multiplying by $s$ rescales every singular value but does not create new nonzero singular values. If $s = 0$, the update becomes the zero matrix, whose rank is even smaller. So in all cases,

$$\operatorname{rank}(\Delta W_{\mathrm{LoRA}}) = \operatorname{rank}(sBA) \le \operatorname{rank}(BA) \le r.$$

This is the rank-bound half of the proof: LoRA is not merely encouraging a low-rank update; it parameterizes the update so that low rank is guaranteed.

The parameter-count argument is just as direct. In full fine-tuning, the update $\Delta W$ has one trainable parameter for every entry of a $d \times k$ matrix:

$$p_{\mathrm{full}} = dk.$$

In LoRA, the original matrix $W_0$ is frozen. The only trainable entries are those in $A$ and $B$. Matrix $A$ contributes $rk$ parameters, and matrix $B$ contributes $dr$ parameters, so

$$p_{\mathrm{LoRA}} = rk + dr = r(k+d).$$

The trainable-parameter fraction is therefore

$$\rho = \frac{p_{\mathrm{LoRA}}}{p_{\mathrm{full}}} = \frac{r(k+d)}{dk}.$$

When $r \ll d,k$, this fraction can be tiny. For square matrices with $d=k=n$, it becomes approximately

$$\rho = \frac{2rn}{n^2} = \frac{2r}{n},$$

so the savings grow as the layer width grows. This is precisely why LoRA becomes more attractive for large models: the full matrix scales quadratically in width, while the factorized update scales linearly in width for fixed $r$.

There is a subtle but important trade-off hiding inside this proof. The low-rank constraint reduces memory, optimizer state, checkpoint size, and communication cost, but it also restricts the set of updates the model can represent. LoRA works well when the task-specific adaptation can be captured by a relatively low-dimensional change to the pretrained computation. It may struggle if the desired update genuinely requires high rank, or if LoRA is inserted into layers that are not responsible for the task-relevant behavior. The theorem does not say low rank is always enough; it says that if low rank is enough, LoRA gives a very efficient way to train it.

The visual below compresses the proof into two linked claims. The first claim is geometric: $BA$ must live inside the column space supplied by $B$, so its rank cannot exceed the bottleneck size $r$. The second claim is arithmetic: instead of paying for all $dk$ entries of a dense update, we pay only for the entries of the two thin factors, $rk + dr$.

Read the final highlighted expressions as the reusable facts we will carry forward:

$$\operatorname{rank}(\Delta W_{\mathrm{LoRA}}) \le r, \qquad \rho = \frac{r(k+d)}{dk}.$$

These two statements are the entire motivation for the LoRA parameterization that comes next: freeze the large pretrained matrix, learn only a small low-rank residual, and rely on the pretrained model to provide the high-dimensional representation that the low-rank update gently redirects.

Having proved that a product $BA$ cannot have rank larger than its inner dimension $r$, we can now turn that fact into a fine-tuning method. The key move in LoRA is not to invent a new kind of neural layer, but to reinterpret full fine-tuning as learning an additive correction to a pretrained weight matrix—and then restrict that correction to a low-rank family.

For a pretrained linear layer, suppose the original weight is

$$W_0 \in \mathbb{R}^{d \times k},$$

mapping an input activation $h \in \mathbb{R}^k$ to an output preactivation $z \in \mathbb{R}^d$. In ordinary full fine-tuning, we update all entries of $W_0$. Equivalently, after training we can write the adapted weight as

$$W = W_0 + \Delta W.$$

This is a useful conceptual shift. Instead of thinking “we train $W$,” we think “we keep the pretrained solution $W_0$ and learn a task-specific displacement $\Delta W$.” Full fine-tuning places essentially no structural constraint on $\Delta W$: it may be any matrix in $\mathbb{R}^{d \times k}$. That expressiveness is powerful, but expensive, because it requires storing and optimizing $dk$ trainable parameters for each adapted matrix.

LoRA asks whether we really need that much freedom. Empirically, many downstream adaptations appear to live in a much smaller “intrinsic” subspace than the full parameter space suggests. So instead of learning an arbitrary dense $\Delta W$, LoRA constrains the update to be low-rank:

$$\Delta W = \Delta W_{\mathrm{LoRA}} = sBA, \qquad s = \alpha/r.$$

Here the two learned matrices have shapes

$$A \in \mathbb{R}^{r \times k}, \qquad B \in \mathbb{R}^{d \times r},$$

where

$$r \ll \min(d,k).$$

The product $BA$ has the same shape as $W_0$, namely $d \times k$, so it can be added directly to the frozen pretrained weight. But because it factors through an $r$-dimensional bottleneck, its rank is at most $r$. This is precisely the rank bound from the previous section made operational: LoRA does not merely hope for a low-rank update; it parameterizes the update so that low rank is guaranteed.

The scalar $s$ is usually written as

$$s=\alpha/r,$$

where $\alpha$ is a scaling hyperparameter. This scaling is not just cosmetic. Since changing $r$ changes the size and behavior of the low-rank branch, $\alpha/r$ helps keep the magnitude of the adaptation reasonably controlled across different ranks. In practice, $\alpha$ gives the practitioner a knob for the strength of the LoRA update, while $r$ controls its capacity.

With this parameterization, the adapted linear layer becomes

$$z = (W_0+sBA)h+b_0.$$

By associativity, we can rewrite the same computation as

$$z = W_0h+sB(Ah)+b_0.$$

This second form is the one that reveals the implementation. We do not need to explicitly materialize the full matrix $sBA$ during training. Instead, the input $h$ first passes through the small matrix $A$, producing an $r$-dimensional intermediate representation $Ah$. Then $B$ maps that small vector back to the output dimension $d$. The result is scaled by $s$ and added in parallel to the frozen pretrained computation $W_0h+b_0$.

The trainable parameters are therefore only

$$\phi=\{A,B\}$$

for each selected weight matrix. The original pretrained parameters $W_0$ and $b_0$ remain frozen. This is the central efficiency gain: instead of training $dk$ parameters for a full dense update, LoRA trains

$$p_{\mathrm{LoRA}} = rk + dr = r(k+d)$$

parameters. When $r$ is small compared with $d$ and $k$, this can be orders of magnitude smaller than full fine-tuning while still allowing a meaningful task-specific displacement.

There are a few subtle assumptions hidden in this elegant construction. LoRA works best when the useful downstream change can be approximated well by a low-rank update in the chosen weight matrices. If the task requires many independent directions of change, a very small $r$ may underfit. Conversely, increasing $r$ improves expressiveness but also increases memory, compute, and the risk of losing the parameter-efficiency advantage. The choice of where to insert LoRA also matters: in transformers, it is commonly applied to attention projections such as query and value matrices, and sometimes to key, output, or MLP projections depending on the task and budget.

The visual that accompanies this derivation condenses the entire idea into one flow: start from the full fine-tuning identity $W=W_0+\Delta W$, replace the unconstrained update $\Delta W$ with the structured low-rank product $sBA$, and then substitute that replacement into the layer’s forward pass. The important conceptual distinction is also encoded visually: $W_0$ and $b_0$ are frozen, while only $A$ and $B$ are trainable.

The matrix-shape sketch is especially useful because it makes the bottleneck concrete. A wide matrix $A$ maps from $k$ dimensions down to $r$, and a tall matrix $B$ maps from $r$ back up to $d$. Their product has the right $d \times k$ shape to behave like a full update, but it is forced to pass through the narrow rank-$r$ channel. That is LoRA in one sentence: full fine-tuning’s additive update, restricted to a scaled low-rank form, with only the factors trained.

Once we have committed to the LoRA parameterization, the next question is operational: what actually happens during the forward pass? The key point is that LoRA does not replace the pretrained layer with an entirely new trainable layer. Instead, it keeps the original computation intact and adds a small trainable correction in parallel.

Start with a frozen affine layer from the pretrained model. If the input activation is $h \in \mathbb{R}^k$, the original layer computes

$$z = W_0 h + b_0,$$

where $W_0 \in \mathbb{R}^{d \times k}$ and $b_0 \in \mathbb{R}^d$. In full fine-tuning, we would directly update $W_0$, turning it into $W_0 + \Delta W$. LoRA instead freezes $W_0$ and restricts the update to a low-rank form:

$$z = (W_0 + \Delta W_{\mathrm{LoRA}})h + b_0,$$

with

$$\Delta W_{\mathrm{LoRA}} = sBA, \qquad s = \frac{\alpha}{r}.$$

Substituting this into the layer gives the forward computation

$$z = W_0h + b_0 + sB(Ah).$$

This equation is the practical heart of LoRA. The model output is the sum of two paths:

• a frozen base path, $h \mapsto W_0h + b_0$, which preserves the pretrained model;
• a trainable LoRA path, $h \mapsto Ah \mapsto B(Ah) \mapsto sB(Ah)$, which learns the task-specific adaptation.

The trainable path is deliberately narrow. If $A \in \mathbb{R}^{r \times k}$, then

$$Ah \in \mathbb{R}^r.$$

So the input activation is first projected down into an $r$-dimensional bottleneck, then expanded back to the layer output dimension by $B \in \mathbb{R}^{d \times r}$. When $r \ll \min(d,k)$, this path has far fewer parameters than a dense $d \times k$ update. Instead of learning every entry of $\Delta W$, LoRA learns only

$$rk + dr = r(k+d)$$

parameters for that layer.

The scaling factor $s = \alpha/r$ is also important. Without scaling, changing the rank $r$ would change the typical magnitude of the LoRA branch, making hyperparameters harder to transfer across ranks. The parameter $\alpha$ acts like a LoRA-specific gain, while division by $r$ normalizes the contribution as the bottleneck width changes. In practice, this helps make the low-rank branch behave like a controlled residual update rather than an unstable perturbation to the frozen model.

A useful way to think about the forward pass is that LoRA is not asking the small matrices $A$ and $B$ to re-learn the original layer. The pretrained matrix $W_0$ already contains the general-purpose representation learned during pretraining. The LoRA branch only needs to learn a directional correction in weight space: a low-dimensional adjustment that nudges the frozen model toward the downstream task.

This also explains why LoRA can fail when the required adaptation is not well captured by a low-rank update. If the downstream task requires many independent changes across the full weight matrix, a small rank $r$ may be too restrictive. But when the task-specific shift is concentrated in a low-dimensional subspace—which is often empirically true for adapting large pretrained models—then the bottleneck is a feature, not a bug. It forces parameter efficiency while still allowing meaningful movement in function space.

Many implementations also apply LoRA dropout to the input of the trainable branch:

$$z = W_0h + b_0 + sBA(M_{\mathrm{drop}}h).$$

The dropout mask affects only the LoRA path, not the frozen base computation. This is a subtle but important design choice: the pretrained model remains deterministic and intact, while the adapter branch is regularized. During training, dropout discourages the LoRA branch from depending too heavily on specific activation coordinates; during inference, the dropout is disabled just as in standard neural network layers.

The visual below compresses this forward pass into a computation graph. The input $h$ splits into two parallel routes: the gray frozen route through $W_0$ and $b_0$, and the blue trainable route through the two small matrices $A$ and $B$. The narrow middle activation $Ah \in \mathbb{R}^r$ makes the low-rank bottleneck explicit.

It is worth reading the diagram as an implementation recipe as much as a mathematical identity. In training, gradients flow through $A$ and $B$, while $W_0$ and $b_0$ remain fixed. At inference time, the two-branch computation can either be evaluated directly as $W_0h + b_0 + sB(Ah)$, or merged into the base weight as $W_0 + sBA$ for a standard single-matrix forward pass.

Now that the forward pass has been reduced to “a frozen linear layer plus a small parallel correction,” the next important question is not mathematical but operational: what exactly is being trained? LoRA is easy to describe as an extra branch, but its practical value comes from treating the pretrained model as a fixed computational substrate while letting only the low-rank adapter carry task-specific change.

For a pretrained weight matrix $W$, the LoRA-augmented layer behaves like

$$h = Wx + \Delta W x,$$

where the update $\Delta W$ is represented indirectly by two small trainable matrices. The key implementation decision is that $W$ remains frozen. It is still used in the forward pass, and it still shapes the activations that later layers see, but the optimizer is not allowed to modify it.

That distinction matters. Freezing $W$ does not mean removing it from the computation. The base model still provides the pretrained function: all of its attention patterns, feature extractors, and learned representations remain active. LoRA simply adds a trainable residual direction on top of that function. In this sense, LoRA fine-tuning is closer to learning a task-specific correction field than relearning the model.

A useful way to think about the training graph is:

• the main path computes the original pretrained transformation;
• the adapter path computes a low-rank residual update;
• the outputs are added;
• gradients update only the adapter parameters.

This is also why LoRA can be parameter-efficient without being computationally invisible. During training, activations still need to flow through the full network, and backpropagation still needs to propagate error signals across layers. What is saved is not the need to run the model, but the need to store and update optimizer state for every pretrained parameter.

That saving is large. In standard full fine-tuning, each parameter typically needs not only its value but also gradients and optimizer statistics, such as Adam’s first and second moment estimates. LoRA avoids this cost for the frozen base weights. The pretrained matrices remain present in memory for inference and forward computation, but they do not accumulate gradients or optimizer state. The trainable state is concentrated in the small adapter matrices.

There is a subtle implementation failure mode here. If we merely “freeze everything” too aggressively—especially by wrapping large portions of the model in a no-gradient context—we can accidentally prevent gradients from reaching adapters in earlier layers. A frozen module should usually mean its parameters do not receive gradients, not that its computation is detached from the graph. The network still needs a differentiable path through the frozen operations so that downstream loss signals can reach upstream LoRA modules.

So the clean mental model is not “turn off the model and train LoRA.” It is:

$$\text{use the pretrained model normally, but optimize only the adapter parameters.}$$

This is the separation that makes LoRA modular. The base model can be shared across tasks, while each task stores only its lightweight adapter. Later, the adapter can be loaded dynamically, swapped for another one, or merged into the base weights for deployment.

The visual below compresses this idea into a single training-flow picture: a large frozen route carries the pretrained computation, while a much smaller trainable route runs in parallel and contributes the learned correction. The important takeaway is the asymmetry: both routes affect the forward output, but only the adapter route is seen by the optimizer.

It also sets up the next issue naturally. If LoRA begins as an additive branch, then its initial behavior must be chosen carefully. Before fine-tuning starts, we usually want the adapted model to behave exactly like the pretrained model—not approximately, but functionally unchanged at step zero. That requirement leads directly to the initialization scheme used for the two LoRA matrices.

Before deciding where to insert LoRA modules in a transformer, it is worth getting one small but crucial detail right: how the LoRA parameters are initialized. This detail is easy to overlook because LoRA is often summarized as “freeze $W_0$, train a low-rank update.” But if the update is initialized carelessly, the model’s behavior can change immediately before any learning has happened, destroying one of the main advantages of adaptation from a pretrained model.

Recall the LoRA parameterization for a frozen linear layer:

$$z = (W_0 + \Delta W_{\mathrm{LoRA}})h + b_0,$$

with

$$\Delta W_{\mathrm{LoRA}} = sBA, \qquad s = \frac{\alpha}{r}.$$

Here $W_0$ and $b_0$ are frozen pretrained parameters, $h$ is the layer input, $A \in \mathbb{R}^{r \times d_{\text{in}}}$, $B \in \mathbb{R}^{d_{\text{out}} \times r}$, and $r$ is the LoRA rank. The scalar $s$ controls the magnitude of the adaptation, often written as $\alpha/r$, so that changing the rank does not automatically make the update scale explode.

The desired behavior at the beginning of training is simple: the LoRA-augmented model should compute exactly the same function as the pretrained model at step zero. That is, before observing any task-specific gradients, we do not want the adapter to perturb the logits, hidden states, or attention projections. The pretrained model is already a highly optimized solution; LoRA should begin as a function-preserving reparameterization, not as random noise injected into a delicate computation graph.

LoRA achieves this by using an asymmetric initialization:

$$A_0 \text{ random}, \qquad B_0 = 0.$$

Then the low-rank update at initialization is exactly zero:

$$\Delta W_{\mathrm{LoRA}} = sB_0A_0 = s\,0\,A_0 = 0 \qquad (t=0).$$

Therefore the layer output is also exactly the pretrained output:

$$z = (W_0+sB_0A_0)h+b_0 = W_0h+b_0 \qquad (t=0).$$

This is stronger than saying the perturbation is “small.” It is not merely small in expectation, nor small under a variance calculation. It is identically zero for every input $h$. The entire network, with LoRA modules inserted, initially represents the same function $f_{\theta_0}$ as the frozen pretrained model.

The asymmetry raises a natural question: if $B_0=0$, have we accidentally blocked learning? The answer is no, but the gradient flow is staged. Let $g_z = \nabla_z \mathcal{L}$ be the gradient arriving at the output of this linear layer. For the LoRA branch,

$$z = W_0h + sBAh + b_0.$$

The gradients with respect to $A$ and $B$ are

$$\nabla_A\mathcal{L} = sB_0^\top g_z h^\top = 0,$$

while

$$\nabla_B\mathcal{L} = s\,g_z(A_0h)^\top,$$

which is generally nonzero as long as $A_0h$ is not always zero and the loss gradient $g_z$ is nonzero.

So the first update goes into $B$, not $A$. This is the important subtlety: random $A_0$ creates nonzero features $A_0h$ for $B$ to learn from, while zero $B_0$ prevents those features from affecting the model output at initialization. After one or more optimization steps, $B_t$ becomes nonzero, and then the gradient to $A$,

$$\nabla_A\mathcal{L} = sB_t^\top g_z h^\top,$$

also becomes nonzero. In other words, $B$ learns first; once $B$ opens the path, $A$ begins to adapt as well.

This also explains why initializing both factors to zero would be a bad idea. If

$$A_0 = 0, \qquad B_0 = 0,$$

then the update still preserves the pretrained function, but now

$$\nabla_B\mathcal{L} = s\,g_z(A_0h)^\top = 0.$$

Both factors would receive zero gradient at step zero, and the LoRA branch would be stuck. Conversely, initializing both $A_0$ and $B_0$ randomly would allow gradients to flow immediately, but it would produce a random nonzero $\Delta W_{\mathrm{LoRA}}$, perturbing the pretrained function before training begins. The LoRA initialization is therefore a careful compromise:

• Random $A_0$: supplies a nondegenerate low-dimensional feature map.
• Zero $B_0$: guarantees $\Delta W_{\mathrm{LoRA}}=0$ at initialization.
• Nonzero $\nabla_B\mathcal{L}$: lets learning begin immediately.
• Delayed $\nabla_A\mathcal{L}$: allows $A$ to learn once $B$ has moved away from zero.

The visual summary emphasizes exactly this “safe start” structure. The frozen pretrained branch carries $h$ through $W_0$ and produces the original contribution $W_0h+b_0$. In parallel, the LoRA branch computes $A_0h$, but because the next factor is $B_0=0$, the entire scaled adaptation $sB_0A_0h$ vanishes. The two branches recombine, yet the output remains

$$z = W_0h+b_0 \qquad (t=0).$$

The key intuition to keep in mind is that LoRA does not begin by changing the model. It begins by adding a trainable path whose output is exactly silent, but whose parameters are arranged so that gradients can wake it up. That is what makes LoRA a stable fine-tuning method rather than just a low-rank random perturbation of a pretrained network.

With the initialization story in place, the next question is not how to add a LoRA branch to one linear layer, but which linear layers should receive one. A transformer block contains many affine maps, and LoRA is not a new kind of transformer layer; it is a local replacement rule applied repeatedly to a chosen set of existing matrices. The practical art is choosing that set so that the adapter has enough expressive power to steer the model, without turning parameter-efficient fine-tuning back into something close to full fine-tuning.

Recall the single-layer LoRA modification. For a frozen pretrained linear map,

$$z = W_0 h + b_0,$$

LoRA keeps $W_0$ and $b_0$ fixed and learns a low-rank residual update:

$$z = W_0h + sBAh + b_0, \qquad \Delta W_{\mathrm{LoRA}} = sBA.$$

Here $A$ projects the input representation into a rank-$r$ adapter subspace, $B$ projects it back to the output dimension, and $s$ is a scaling factor, often written as $\alpha/r$. The key point is that this rule is matrix-local: wherever the transformer has a linear map $W_0$, we can decide independently whether to attach a LoRA branch.

In a self-attention block, the most visible candidates are the four attention projections. Given hidden states $H \in \mathbb{R}^{m \times d_{\mathrm{model}}}$, the model forms queries, keys, and values through learned projections $W_Q, W_K, W_V$, then applies another projection $W_O$ after the attention aggregation. Abstractly, LoRA asks us to choose a target set

$$\mathcal{S}\subseteq\{W_Q,W_K,W_V,W_O\} \quad\text{plus selected feed-forward matrices.}$$

For each $W_0 \in \mathcal{S}$, we replace the frozen map by the LoRA-augmented one:

$$z = W_0h+b_0 \quad\leadsto\quad z = W_0h+sBAh+b_0.$$

For matrices outside $\mathcal{S}$, nothing changes: they remain exactly frozen and receive no trainable low-rank branch.

This choice matters because different projections control different aspects of the computation. Roughly speaking, $W_Q$ influences what each token asks for, $W_K$ influences how tokens are matched, $W_V$ influences what information is carried forward once attention weights are chosen, and $W_O$ determines how the attended information is mixed back into the residual stream. Feed-forward matrices, meanwhile, often carry much of the model’s nonlinear feature transformation capacity. Adapting these locations gives LoRA different routes for changing behavior.

The original LoRA paper found that adapting only queries and values,

$$\mathcal{S}=\{W_Q,W_V\},$$

was often a strong default. This is intuitively appealing: modifying queries changes the attention pattern, while modifying values changes the content being routed. However, this is not a universal law. Many modern fine-tuning recipes adapt more projections, such as $W_Q,W_K,W_V,W_O$, and sometimes the MLP up/down/gate projections as well. The larger the target set, the more expressive the adapter becomes—but also the more parameters, memory traffic, and possible overfitting pressure it introduces.

A useful way to think about the design space is:

• Fewer target matrices: cheaper, more compact, often surprisingly effective, but may underfit tasks requiring deeper representational change.
• More target matrices: more flexible and often stronger on difficult instruction, reasoning, or domain-shift tasks, but less parameter-efficient.
• Attention-only LoRA: directly changes routing and information selection.
• Attention + MLP LoRA: also changes the tokenwise feature transformations inside each block.

There is also a subtle failure mode: if $\mathcal{S}$ is too restrictive, increasing the rank $r$ may not solve the problem. A high-rank update to the “wrong” matrices can still be less useful than a modest-rank update to the matrices where the task needs control. Placement and rank interact. LoRA’s low-rank assumption is about the needed update at a chosen location; it does not say that every downstream adaptation can be well represented by perturbing only one or two projections.

The visual below consolidates this idea as a transformer attention block with LoRA branches attached only to selected frozen projections. The gray boxes represent pretrained matrices that remain fixed. The blue side branches represent the trainable low-rank path $sBAh$, which runs in parallel with the frozen map and is added back to its output.

Notice especially the highlighted $W_Q$ and $W_V$ projections: they mark the common original LoRA default. The same construction could be extended to $W_K$, $W_O$, or feed-forward matrices by adding identical low-rank side branches there too. In other words, LoRA placement is not a new derivation each time—it is the repeated application of the same constrained update rule over a chosen target set $\mathcal{S}$.

Once we know where LoRA is typically inserted in a Transformer—often in attention projections such as $W_q$, $W_v$, and sometimes $W_o$ or MLP matrices—the next question is more fundamental: why should a low-rank update be a reasonable restriction at all? Full fine-tuning allows every entry of a weight matrix to move independently. LoRA, by contrast, insists that the learned update must factor through a small bottleneck dimension $r$. That is a strong constraint, so we need a mathematical reason to believe it can preserve the important part of adaptation.

Recall the contrast. In full fine-tuning, a pretrained weight matrix $W\in\mathbb{R}^{d\times k}$ is changed to

$$W+\Delta W,$$

where $\Delta W$ can be any matrix of the same shape. LoRA freezes $W$ and learns only a structured update

$$\Delta W_{\mathrm{LoRA}} = sBA,$$

where $B\in\mathbb{R}^{d\times r}$, $A\in\mathbb{R}^{r\times k}$, and $s$ is a scalar scaling factor. Because this update passes through an $r$-dimensional inner space, its rank is limited:

$$\operatorname{rank}(\Delta W_{\mathrm{LoRA}}) = \operatorname{rank}(sBA) \le r.$$

So LoRA is not merely “using fewer parameters.” It is asserting that the useful task-specific movement in weight space can be expressed, or at least well-approximated, by a matrix whose action lives in only a few dominant directions.

The relevant theorem is the Eckart–Young–Mirsky theorem, specialized here to the Frobenius norm. Let $M\in\mathbb{R}^{d\times k}$ be any matrix, with singular value decomposition

$$M = U\Sigma V^\top,$$

where the singular values satisfy

$$\sigma_1\ge \sigma_2\ge \sigma_3\ge \cdots \ge 0.$$

The singular values measure how much “energy” or variation the matrix has along orthogonal input-output directions. Large singular values correspond to directions in which the matrix has a strong effect; small singular values correspond to weaker directions.

The theorem says that if we are allowed to approximate $M$ using only a matrix of rank at most $r$, then the best possible approximation in Frobenius norm is obtained by keeping only the top $r$ singular values and discarding the rest:

$$M_r = U\operatorname{diag}(\sigma_1,\ldots,\sigma_r,0,\ldots)V^\top.$$

Moreover, the approximation error is exactly the energy of the discarded tail:

$$\|M-M_r\|_F^2 = \sum_{i>r}\sigma_i^2.$$

This is the key bridge to LoRA. Suppose we imagine performing full fine-tuning and obtaining some update $\Delta W$. If the singular values of $\Delta W$ decay quickly, then most of the update’s Frobenius energy is concentrated in a small number of directions. In that case, a rank-$r$ approximation may capture nearly all of the meaningful movement:

$$\Delta W \approx (\Delta W)_r.$$

LoRA does not explicitly compute the SVD of the full fine-tuning update during training. Instead, it directly optimizes a low-rank factorization $BA$. But the theorem tells us why this is a plausible parameterization: if the true useful update is approximately low-rank, then there exists a low-rank matrix that is close to it.

There is an important subtlety here. The theorem is an approximation theorem, not a guarantee that LoRA will always match full fine-tuning. It says that low-rank approximation works well when the spectrum decays. If the desired update has many equally important singular directions, then the tail energy

$$\sum_{i>r}\sigma_i^2$$

may remain large, and a small-rank LoRA adapter may underfit. This is one reason the choice of rank $r$, target modules, dataset size, and task difficulty all matter in practice.

The theorem also clarifies what LoRA is betting on:

• Good case: the full update has a few dominant singular directions, so a small $r$ captures most of the useful change.
• Hard case: the full update is high-rank with a flat singular spectrum, so truncating to rank $r$ discards substantial information.
• Practical compromise: many adaptation tasks appear to require only a small subspace of changes, especially when starting from a strong pretrained model.

The visual below compactly organizes this logic. The upper equation emphasizes the rank constraint imposed by the LoRA factorization $sBA$. The theorem block then states the optimal low-rank approximation result: among all rank-$r$ matrices, truncated SVD gives the closest approximation in Frobenius norm.

The singular-value spectrum on the side is the conceptual heart of the result. The first $r$ bars represent the directions retained by the rank-$r$ approximation, while the gray tail represents discarded energy. When that tail is small, LoRA’s low-rank restriction is not a severe limitation; it is an efficient way to focus learning on the dominant adaptation directions.

The previous result told us what the optimal rank-$r$ approximation is: keep the top $r$ singular values and discard the rest. But for LoRA, the more important lesson is why this is true. LoRA will soon restrict a weight update $\Delta W$ to the form $BA$, which automatically has rank at most $r$. So before we turn this into an algorithm, we want to understand the geometry of the constraint: if you are only allowed $r$ low-rank degrees of freedom, where should you spend them?

Let $M$ be the matrix we want to approximate, with singular value decomposition

$$M = U \Sigma V^\top,$$

where $\Sigma$ is diagonal with singular values $\sigma_1 \ge \sigma_2 \ge \cdots \ge 0$. We compare $M$ against an arbitrary rank-$r$ candidate, written in factored form as $BA$. This notation is deliberately reminiscent of LoRA: if $B \in \mathbb{R}^{d_{\text{out}} \times r}$ and $A \in \mathbb{R}^{r \times d_{\text{in}}}$, then

$$\operatorname{rank}(BA) \le r.$$

The proof begins with a change of coordinates. Instead of measuring the approximation error directly in the original basis, rotate both sides into the singular-vector coordinates of $M$. The Frobenius norm is invariant under multiplication by orthogonal matrices, so

$$\|M-BA\|_F^2 = \|U^\top(M-BA)V\|_F^2 = \|\Sigma - U^\top BA V\|_F^2.$$

This step is powerful because it turns the target matrix into something extremely simple: a diagonal matrix $\Sigma$. All the complexity of $M$'s left and right singular vectors has been absorbed into the coordinate system. Meanwhile, the candidate approximation becomes $U^\top BA V$. It may no longer look like a clean product $BA$, but its rank cannot increase under orthogonal rotations:

$$\operatorname{rank}(U^\top BA V) \le \operatorname{rank}(BA) \le r.$$

So the problem has been reduced to a more transparent one: approximate a diagonal matrix $\Sigma$ using any matrix of rank at most $r$. In these coordinates, each diagonal entry $\sigma_i$ represents the amount of energy in an independent singular direction. The Frobenius error adds squared entrywise errors, so leaving singular value $\sigma_i$ unmatched costs $\sigma_i^2$.

The key intuition is that a rank-$r$ matrix can only fully capture $r$ independent directions. It cannot independently match all diagonal entries of $\Sigma$ if there are more than $r$ nonzero singular values. Therefore, the best strategy is obvious once we are in the right basis: spend the limited rank budget on the directions with the largest singular values, because those are the most expensive to ignore.

Formally, every rank-$r$ candidate must incur at least the squared tail energy of the singular values it cannot represent:

$$\|\Sigma - U^\top BA V\|_F^2 \ge \sum_{i>r} \sigma_i^2.$$

This inequality is the heart of the theorem. It says that no clever choice of $B$ and $A$, no off-diagonal mixing, and no alternative coordinate trick can beat the tail energy left behind after the top $r$ singular directions are accounted for. The singular-vector basis is already the coordinate system in which the approximation problem is most clearly decomposed.

Equality is achieved by the truncated SVD approximation

$$M_r = U \operatorname{diag}(\sigma_1,\ldots,\sigma_r,0,\ldots) V^\top,$$

for which

$$\|M-M_r\|_F^2 = \sum_{i>r}\sigma_i^2.$$

So the lower bound is not merely a pessimistic guarantee; it is tight. Truncated SVD attains the best possible Frobenius error among all rank-$r$ matrices.

This proof also clarifies a subtle but important point for LoRA. LoRA is not usually computing the truncated SVD of an optimal full update during training. Instead, it directly learns a low-rank update $BA$. But the SVD theorem tells us why such a parameterization can be reasonable: if the task-specific update mostly lives in a small number of dominant directions, then a low-rank factorization can capture most of its useful energy while using far fewer parameters.

The visual below compresses this argument into the essential chain: rotate into singular-vector coordinates, observe that the candidate still has rank at most $r$, and then see the diagonal singular values split into “kept” directions and the unavoidable residual tail. The blue entries correspond to the rank budget being spent on the largest singular directions; the gray tail corresponds to the error term $\sum_{i>r}\sigma_i^2$.

Read this picture as the bridge from linear algebra to LoRA: once updates are constrained to rank $r$, the best possible use of that rank is to align with the most important directions of change. LoRA turns that mathematical constraint into a trainable parameterization.

Having argued that a best low-rank approximation is meaningful in the SVD sense, we can now turn that linear-algebra fact into an actual training algorithm. LoRA is not a mysterious new optimizer or a different learning objective. It is ordinary supervised fine-tuning with one structural constraint: instead of updating the pretrained weight matrix $W_0$, we freeze it and learn a low-rank additive correction.

For a target linear map, suppose the pretrained layer computes

$$z = W_0 h + b_0,$$

where $h$ is the input activation to that layer. Full fine-tuning would replace $W_0$ by $W_0 + \Delta W$, with $\Delta W$ unconstrained and the same shape as $W_0$. LoRA restricts that update to factor through a small rank-$r$ bottleneck:

$$\Delta W = sBA, \qquad s = \frac{\alpha}{r}.$$

So the adapted layer becomes

$$z = W_0h + sBAh + b_0.$$

Here $A \in \mathbb{R}^{r \times d_{\text{in}}}$ projects the input activation into a low-dimensional adaptation space, and $B \in \mathbb{R}^{d_{\text{out}} \times r}$ maps it back to the output dimension. The pretrained matrix $W_0 \in \mathbb{R}^{d_{\text{out}} \times d_{\text{in}}}$ remains fixed throughout training.

The scaling factor $s=\alpha/r$ is a practical normalization convention. Since changing the rank $r$ changes the number of paths through the low-rank adapter, the scale $\alpha/r$ helps keep the magnitude of the update reasonably controlled as $r$ varies. In practice, $\alpha$ is often treated as a LoRA hyperparameter analogous to an adapter strength.

Training then minimizes the usual empirical risk over the trainable LoRA parameters $\phi$, which collect all inserted $A,B$ factors:

$$\mathcal{L}(\phi) = \frac{1}{n} \sum_{i=1}^{n} \ell(\hat{y}_i,y_i).$$

The important distinction is not the loss, but the parameter set receiving optimizer updates. Gradients still flow through the network computational graph, because the LoRA factors may be deep inside a transformer block. But the optimizer state is allocated only for $A$ and $B$, not for the original pretrained parameters $\theta_0$.

A common initialization is asymmetric:

• initialize $A$ randomly,
• initialize $B=0$,
• freeze $W_0$ and $b_0$.

This makes the initial LoRA update exactly zero:

$$sBA = 0,$$

so the model begins training with precisely the pretrained function. That is a useful stability property: before seeing any task data, LoRA has not perturbed the base model at all. There is one subtle consequence: at the very first step, the gradient into $A$ is zero because it is multiplied by $B^\top$. The gradient into $B$, however, is generally nonzero because $A$ is random. After $B$ moves away from zero, both factors can train normally.

The algorithm also depends on a choice of target modules $\mathcal{S}$. LoRA is usually not inserted into every matrix in the model. In transformer language models, common targets include attention projection matrices such as query and value projections, and sometimes key, output, or MLP projection matrices. This choice is a systems and modeling trade-off:

• targeting more matrices increases adaptation capacity,
• targeting fewer matrices reduces memory and compute overhead,
• attention projections often give strong performance per parameter,
• very small ranks may underfit tasks requiring broad behavioral change.

The core training loop is therefore simple. Freeze the pretrained model, attach low-rank trainable branches to selected matrices, compute predictions using the modified layer rule, evaluate the supervised loss, and update only the LoRA factors with Adam or another optimizer. The base model participates in the forward computation, but it does not accumulate trainable updates.

This is why LoRA is best understood as constrained fine-tuning, not as prompt tuning or retrieval augmentation. The model’s internal computations are genuinely modified, but only through a low-rank subspace of possible weight updates. Compared with full fine-tuning, LoRA dramatically reduces trainable parameters and optimizer memory. Compared with purely external methods, it still changes the effective network function in a task-specific way.

The visual below condenses this into the algorithmic skeleton: first freeze $\theta_0$, then initialize one pair of low-rank factors for each selected target matrix $W_0 \in \mathcal{S}$, then run the ordinary supervised optimization loop. The highlighted layer equation

$$z = W_0h + sBAh + b_0$$

is the entire mechanism: the frozen pretrained path remains intact, while the trainable low-rank path learns the task-specific residual.

The key takeaway is that LoRA fine-tuning does not require a new learning objective or special gradient estimator. It is standard backpropagation with a carefully restricted parameterization. All of the efficiency gains follow from the fact that Adam moments, gradients, and trainable weights are maintained only for the small matrices $A$ and $B$, while the large pretrained matrices stay fixed.