Variational Autoencoders: Principles, Derivations, and Applications

Before we can understand what a variational autoencoder is optimizing, we need to be precise about the problem it is trying to solve. A VAE is not merely an “autoencoder with noise.” It is a generative model: a model whose goal is to learn something about the probability distribution that produced the observed data.

Suppose we are given a dataset

$$\mathcal{D}=\{x^{(1)},\ldots,x^{(N)}\},$$

where each observation $x^{(n)}\in\mathbb{R}^D$. For images, $D$ might be the number of pixels; for audio, it might be waveform samples or spectrogram coefficients; for text embeddings, it might be the embedding dimension. The generative modeling problem is to learn a distribution $p_\theta(x)$ that approximates the unknown data-generating distribution $p_{\text{data}}(x)$.

This goal has three closely related interpretations. First, we may want density estimation: given a new point $x$, how likely is it under the model? That is, can we evaluate or approximate $p_\theta(x)$? Second, we may want synthesis: can we draw new samples $x\sim p_\theta(x)$ that look like plausible members of the dataset, but are not exact copies? Third, we may want representation learning: can the model discover compact, structured variables that explain meaningful factors of variation in the data?

These goals are connected, but they are not identical. A model may produce sharp-looking samples while assigning poor likelihoods, or it may achieve good likelihood while generating visually mediocre samples. VAEs are especially interesting because they try to tie these goals together through a probabilistic latent-variable framework: they define a likelihood, support sampling, and learn internal representations.

A first naive idea is to estimate the density directly from the observed data. For example, kernel density estimation places a small “bump” of probability mass around each training example:

$$\hat{p}(x) = \frac{1}{N} \sum_{n=1}^{N} \mathcal{K}\!\left(x-x^{(n)}\right).$$

In low dimensions, this can work surprisingly well. If the data points densely cover the relevant region of space, then nearby kernels overlap and form a smooth estimate of the distribution. But in high dimensions, this intuition breaks down catastrophically.

The reason is the curse of dimensionality. The volume of a ball of radius $r$, or more generally the number of distinguishable regions in a $D$-dimensional space, grows roughly exponentially with $D$. For MNIST, a $28\times 28$ grayscale image lives in $\mathbb{R}^{784}$. Even $N=60{,}000$ images are vanishingly sparse in such a space. Almost every possible pixel vector is not a digit at all; it is visual noise. Worse, many distance-based intuitions fail: in very high dimensions, points tend to become nearly equidistant, making “nearest neighbors” and local smoothing much less reliable.

So the issue is not just that we need more data. The ambient space $\mathbb{R}^D$ is overwhelmingly large, and the data occupies only a tiny, highly structured subset of it. A kernel density estimator spreads probability around observed examples in the ambient space, but most of that space is irrelevant. Unless the kernel bandwidth is extremely small, it assigns mass to unrealistic regions; if the bandwidth is extremely small, it assigns nearly zero density almost everywhere. Either way, the method fails to capture the actual structure of the data distribution.

This motivates the data manifold hypothesis. Although observations may be represented as vectors in a high-dimensional ambient space, the meaningful variation in the data often has much lower intrinsic dimension. For example, handwritten digits vary by stroke thickness, slant, rotation, identity, local style, and other factors. These factors are far fewer than 784 independent pixel degrees of freedom. Informally, the data may lie near a low-dimensional manifold embedded in the high-dimensional observation space.

This is where latent variables enter the story. Instead of modeling $x\in\mathbb{R}^D$ directly as an arbitrary point in the ambient space, we introduce a lower-dimensional variable

$$z\in\mathbb{R}^K, \qquad K\ll D,$$

and imagine that observations are generated from these latent coordinates. The latent variable $z$ should capture the compact explanatory factors, while the model maps from latent space into data space. This does not mean the true data manifold is literally linear, smooth everywhere, or exactly $K$-dimensional. Rather, it is a modeling assumption: useful data distributions often have enough low-dimensional structure that exploiting it is far better than treating all directions in $\mathbb{R}^D$ equally.

The visual below condenses this motivation into a geometric picture. On the left, the high-dimensional ambient space is mostly empty: even many training examples provide negligible coverage when $D$ is large, which is why direct kernel-style density estimation becomes ineffective. On the right, the same data becomes much more intelligible when viewed as concentrated near a lower-dimensional manifold.

The key transition is from “estimate density everywhere in $\mathbb{R}^D$” to “model how low-dimensional latent structure gives rise to high-dimensional observations.” That transition is the conceptual starting point for latent variable models, and it is exactly the path that will lead us to variational autoencoders.

The previous discussion framed generative modeling as a density-learning problem: we want a model that assigns high probability to realistic data and can also produce new samples. But high-dimensional observations—images, audio, text embeddings—rarely vary freely in all ambient dimensions. A handwritten digit image may have thousands of pixels, yet much of its variation can be described by a smaller set of factors: which digit it is, how thick the stroke is, whether it is tilted, how centered it is, and so on. Latent variable models make this intuition explicit.

The core assumption is that each observed data point $x$ is generated from an unobserved, lower-dimensional variable $z$. We do not get to see $z$ in the dataset; it is a hidden explanation for the observation. Instead of modeling the distribution over $x$ directly, we define a two-step generative process:

$$z \sim p(z) = \mathcal{N}(0, I), \qquad x \sim p_{\theta}(x \mid z).$$

Here, $p(z)$ is a simple prior over latent codes, usually chosen to be a standard Gaussian. The conditional distribution $p_{\theta}(x \mid z)$ is the decoder or generative model: given a latent code, it describes a distribution over possible observations. In modern VAEs, this conditional distribution is parameterized by a neural network $f_{\theta}(z)$, which maps latent coordinates into the parameters of a likelihood over data space.

This setup separates two kinds of complexity. The prior $p(z)$ is deliberately simple: sampling from $\mathcal{N}(0,I)$ is easy, and its geometry is well behaved. The decoder carries the burden of learning how simple latent variation becomes rich observed structure. For example, in an idealized MNIST model, nearby values of $z$ might correspond to visually similar digits, while different directions in latent space could control properties such as digit identity, stroke width, or tilt.

The probability assigned to an observation $x$ is obtained by considering all possible latent explanations that could have generated it. This gives the marginal likelihood, also called the evidence:

$$p_{\theta}(x) = \int_{\mathcal{Z}} p_{\theta}(x \mid z)\, p(z)\, dz.$$

This integral is the mathematical heart of latent variable modeling. It says: to evaluate how likely $x$ is, average the likelihood $p_{\theta}(x \mid z)$ over every possible latent code $z$, weighted by how plausible that code was under the prior. A particular $z$ might reconstruct $x$ very well, but if that $z$ lies in an extremely unlikely region of the prior, its contribution is limited. Conversely, common latent codes contribute more, but only if the decoder can plausibly produce $x$ from them.

This is why latent variable models are so appealing. They offer a structured way to generate data:

• sample a simple latent code $z$,
• pass it through a learned decoder,
• obtain a complex observation $x$.

They also offer a conceptual interpretation of representation learning: the model is encouraged to organize meaningful variation in data through the latent space. If the learned representation is smooth, moving through $z$-space should produce coherent changes in generated samples rather than abrupt jumps.

But the same integral that makes the model principled also creates the central computational difficulty. When $p_{\theta}(x \mid z)$ is parameterized by a neural network, the integral

$$\int_{\mathcal{Z}} p_{\theta}(x \mid z)\, p(z)\, dz$$

usually has no closed form. The decoder can be highly nonlinear, and the latent space may have many dimensions. Direct numerical integration becomes infeasible, and naïve Monte Carlo estimates can be too noisy or inefficient for maximum likelihood training. Therefore, although the model defines $p_{\theta}(x)$ formally, we cannot usually evaluate or maximize $\log p_{\theta}(x)$ directly.

This is the key tension that motivates variational inference and, ultimately, the VAE objective. We have a clean generative story and a meaningful likelihood, but the marginalization over hidden causes is intractable. The rest of the VAE framework can be understood as a way to optimize a tractable surrogate for this inaccessible log-likelihood.

The visual below compactly summarizes this idea. The left side uses a graphical-model view: a latent variable $z$ is drawn from the prior, then an observed variable $x$ is generated through $p_{\theta}(x \mid z)$, repeated independently across data points. The shaded observed node emphasizes that $x$ is in the dataset, while $z$ remains hidden.

The right side grounds the abstraction in an MNIST-style example: a latent code can be thought of as controlling semantic or stylistic factors that the decoder turns into an image. The important warning is the intractable integral over $z$: even though the sampling story is simple, evaluating the probability of a given observation requires summing over all latent explanations, which is precisely what we cannot do directly with a neural decoder.

Having introduced latent variable models, we now run into the first serious computational obstacle: the model is easy to write down, but hard to fit. The whole appeal was to define a simple prior $p(z)$, pass $z$ through a decoder, and obtain a flexible distribution over observations $x$. But maximum likelihood training asks us to evaluate

$$p_{\theta}(x) = \int p_{\theta}(x \mid z)\,p(z)\,dz,$$

and that integral is exactly where the trouble begins. For simple decoders, the integral and the posterior over latents may be analytically tractable. For neural network decoders, they usually are not.

The classical tool for latent variable models is Expectation-Maximization. EM alternates between inferring the latent posterior under the current parameters and then updating the parameters using expectations under that posterior. The E-step requires

$$p_{\theta}(z \mid x) = \frac{p_{\theta}(x \mid z)\,p(z)}{p_{\theta}(x)}.$$

This equation is innocent-looking but deceptive. The denominator $p_{\theta}(x)$ is the very marginal likelihood integral we were trying to avoid. In models with conjugacy or linear-Gaussian structure, the posterior has a known form and EM is elegant. But once the decoder becomes nonlinear, the posterior can become highly warped, multimodal, and unavailable in closed form.

A useful contrast is probabilistic PCA. Suppose

$$p(z)=\mathcal{N}(0,I), \qquad p_{\theta}(x \mid z)=\mathcal{N}(Wz+b,\sigma^2 I).$$

Because everything is linear and Gaussian, the posterior $p_{\theta}(z \mid x)$ is also Gaussian. EM can compute its mean and covariance exactly. The latent space remains geometrically well-behaved: observing $x$ carves out an elliptical Gaussian belief over possible $z$'s.

Now replace the linear map $Wz+b$ with a neural network $f_{\theta}(z)$:

$$p_{\theta}(x \mid z)=\mathcal{N}(f_{\theta}(z),\sigma^2 I).$$

The prior is still simple, and the observation noise may still be Gaussian, but the posterior is no longer Gaussian. The inverse image of a given observation $x$ under $f_{\theta}$ may contain many disconnected regions. Different latent codes can decode to similar observations. The posterior may have sharp ridges, separated modes, and strong nonlinear dependencies between latent dimensions. In other words, the generative direction $z \mapsto x$ may be easy to evaluate, while the inference direction $x \mapsto z$ is hard.

One tempting fallback is MAP inference: instead of representing the whole posterior, choose the most likely latent point,

$$\hat{z} = \arg\max_z \log p_{\theta}(z \mid x).$$

This can be useful in some settings, but it is not a satisfactory replacement for posterior inference. A point estimate throws away uncertainty. If the posterior has several plausible modes, MAP picks one and ignores the rest. It also introduces an inner optimization problem for every datapoint, which is expensive during training. Worse, if we need gradients through that optimization procedure, the computation becomes cumbersome and brittle, especially when latent variables are discrete or when the optimization landscape is poorly conditioned.

Another seemingly generic solution is Monte Carlo EM. We might sample latent candidates from the prior,

$$z^{(l)} \sim p(z),$$

and weight them according to how well they explain $x$:

$$w^{(l)} \propto p_{\theta}(x \mid z^{(l)}).$$

This is importance sampling with the prior as the proposal distribution. The problem is that the prior is usually a terrible proposal for the posterior. In high dimensions, most samples from $p(z)$ land in regions that explain a specific $x$ extremely poorly. A tiny number of samples may receive almost all the probability mass, while the rest contribute essentially nothing.

This phenomenon is often called importance weight collapse. As the latent dimension $K$ grows, the effective number of useful samples can collapse toward one:

$$\text{effective sample size} \approx 1 \qquad \text{as } K \to \infty.$$

The phrase “effective sample size” is important: even if we draw thousands of samples, the estimator may behave as though it had only one meaningful sample. This creates exponentially high variance estimates of the E-step quantities and makes naive Monte Carlo learning impractical for expressive latent-variable models.

So the failure is not that EM, MAP, or Monte Carlo are conceptually wrong. Each is reasonable under the right assumptions. The failure is a mismatch between those assumptions and neural generative models:

• Exact EM needs a tractable posterior.
• MAP replaces uncertainty with a single point.
• Prior-based Monte Carlo wastes samples in high-dimensional latent spaces.
• Neural decoders make the true posterior complex and hard to normalize.

The visual below compresses this comparison into the key geometric intuition. In the linear-Gaussian case, posterior inference is clean: the posterior over $z$ is a single Gaussian-shaped region, and EM can proceed exactly. In the nonlinear case, the posterior becomes irregular and multimodal, so the E-step no longer has a closed-form solution.

The weight-collapse sketch at the bottom emphasizes why simply sampling many $z$'s from the prior does not rescue us. In high-dimensional latent spaces, almost all prior samples are irrelevant for a particular observation $x$, and the importance weights concentrate on one lucky sample. This is the motivation for the next idea: instead of solving a separate hard inference problem from scratch for every datapoint, VAEs introduce amortized approximate inference—a learned encoder that predicts an approximate posterior directly.

The failure of direct EM or per-example MAP inference points us toward a different compromise: instead of solving a fresh optimization problem for every datapoint, we will learn a function that performs inference. This is the central move in a variational autoencoder. A VAE is not just “an autoencoder with noise”; it is a probabilistic latent-variable model equipped with a trainable approximation to the posterior.

The generative story begins with a latent variable $z$, typically chosen to live in a relatively low-dimensional continuous space:

$$p(z) = \mathcal{N}(0, I), \qquad z \in \mathbb{R}^K.$$

This distribution is called the prior. It says what kinds of latent codes are plausible before seeing any data. The standard Gaussian prior is not chosen because we believe the true hidden factors of images, text, or molecules are literally independent standard normal variables. Rather, it is chosen because it gives us a simple, smooth, sampleable reference distribution. Later, when we generate new data, we will sample $z \sim p(z)$ and decode it into an observation.

The second ingredient is the decoder, also called the generative model or likelihood model. It specifies how an observed datapoint $x$ is produced from a latent code $z$:

$$p_{\theta}(x \mid z).$$

In a neural VAE, this conditional distribution is parameterized by a neural network $f_{\theta}(z)$. The network does not directly output “the reconstruction” in a purely deterministic sense; rather, it outputs the parameters of a probability distribution over possible observations. For continuous data, a common choice is an isotropic Gaussian likelihood,

$$p_{\theta}(x \mid z) = \mathcal{N}\!\left(f_{\theta}(z),\, \sigma^2 I\right),$$

where $f_{\theta}(z)$ is the mean of the conditional distribution. For binary data, such as binarized MNIST pixels, a common choice is a Bernoulli likelihood,

$$p_{\theta}(x \mid z) = \mathrm{Bernoulli}\!\left(\mathrm{sigmoid}(f_{\theta}(z))\right).$$

Together, the prior and decoder define the actual generative model:

$$p_{\theta}(x, z) = p_{\theta}(x \mid z)\,p(z).$$

This joint distribution is the model’s claim about how data and latent variables co-occur. If we could compute the posterior exactly,

$$p_{\theta}(z \mid x) = \frac{p_{\theta}(x \mid z)p(z)}{p_{\theta}(x)},$$

then inference would be straightforward: given an observed $x$, infer which latent codes $z$ plausibly generated it. But the denominator,

$$p_{\theta}(x) = \int p_{\theta}(x \mid z)p(z)\,dz,$$

is generally intractable for a nonlinear neural decoder. This is the same obstacle we encountered when trying to use exact EM: the posterior is the object we need, but it is not available in closed form.

The VAE introduces a third distribution to address this: the encoder, or approximate posterior,

$$q_{\phi}(z \mid x) = \mathcal{N}\!\left( \mu_{\phi}(x), \mathrm{diag}(\sigma_{\phi}(x)^2) \right).$$

Here another neural network, often denoted $g_{\phi}(x)$, maps an input datapoint to the parameters of a Gaussian distribution over latent codes:

$$g_{\phi}(x) \longrightarrow \left(\mu_{\phi}(x), \sigma_{\phi}(x)\right).$$

This is an approximation to $p_{\theta}(z \mid x)$, not part of the generative model itself. The generative model is still $p(z)p_{\theta}(x \mid z)$. The encoder is an inference mechanism: it gives us a tractable distribution from which we can sample latent codes likely to explain $x$.

The key innovation is amortized inference. In classical variational inference, we might introduce separate variational parameters for each datapoint, for example $q_{\lambda_n}(z \mid x^{(n)})$ with its own $\lambda_n$. That would mean every new datapoint requires its own inference optimization. VAEs instead use a single shared network $q_{\phi}(z \mid x)$ for all datapoints:

$$x^{(n)} \mapsto \left(\mu_{\phi}(x^{(n)}), \sigma_{\phi}(x^{(n)})\right).$$

The cost of inference is therefore amortized over the dataset. We pay once to learn $\phi$, and then inference for a new $x$ is just a forward pass. This is why VAEs scale well to large datasets and why they look architecturally like autoencoders: data goes through an encoder into a latent representation, then through a decoder back into data space. But probabilistically, the encoder and decoder play asymmetric roles.

There is also an important modeling assumption hidden in the standard encoder form. By choosing

$$q_{\phi}(z \mid x) = \mathcal{N}\!\left( \mu_{\phi}(x), \mathrm{diag}(\sigma_{\phi}(x)^2) \right),$$

we assume the approximate posterior is Gaussian with diagonal covariance. This makes sampling and KL-divergence computations convenient, but it can be restrictive. The true posterior $p_{\theta}(z \mid x)$ may be multimodal, skewed, or highly correlated across latent dimensions. Much of the behavior of VAEs—including some failure modes we will discuss later—comes from the tension between a flexible neural decoder and a relatively simple approximate posterior family.

The three distributions therefore have distinct responsibilities:

• Prior $p(z)$: defines the latent space we can sample from.
• Decoder $p_{\theta}(x \mid z)$: defines how latent codes generate observations.
• Encoder $q_{\phi}(z \mid x)$: approximates the intractable posterior for efficient inference.

The visual below consolidates this separation by showing two complementary paths. The generative path starts from $z \sim p(z)$ and moves downward through $p_{\theta}(x \mid z)$ to produce $x$. The inference path goes in the opposite direction: given $x$, the encoder $q_{\phi}(z \mid x)$ produces a distribution over plausible latent codes.

The most important detail to keep in mind is that the right-hand path is not a separate model of the data. It is a learned approximation used to make training and inference tractable. The shared parameter vector $\phi$ is what makes the inference procedure amortized: instead of optimizing a new posterior approximation for every $x^{(n)}$, the VAE learns one encoder network that serves the entire dataset.

Now that we have separated the VAE into its three distributions—the prior $p(z)$, the decoder $p_\theta(x \mid z)$, and the encoder-like approximation $q_\phi(z \mid x)$—we can ask the most basic statistical question: what objective should train the generative model? If the decoder is supposed to define a probability model over observations, then the natural answer is maximum likelihood. We want parameters $\theta$ that assign high probability to the observed dataset:

$$\max_{\theta} \sum_{n=1}^{N} \log p_{\theta}(x^{(n)}).$$

This is the same principle used in ordinary probabilistic modeling: choose the model under which the data would have been most likely. The complication is that in a latent-variable model, $p_\theta(x)$ is not given directly. It is the probability of observing $x$ after averaging over all possible latent explanations $z$.

For a single datapoint, the marginal likelihood, also called the evidence, is

$$p_\theta(x) = \int_{\mathcal{Z}} p_\theta(x \mid z)\,p(z)\,dz,$$

so the log-evidence is

$$\log p_{\theta}(x) = \log \int_{\mathcal{Z}} p_{\theta}(x \mid z)\,p(z)\,dz.$$

Conceptually, this integral says: sample a latent code $z$ from the prior, decode it into a distribution over $x$, and average the probability assigned to the observed $x$ across all possible latent codes. If many plausible latent codes explain $x$, the evidence should be high. If almost no latent code decodes near $x$, the evidence should be low.

The problem is that this integral is almost never analytically tractable for a neural decoder. In a simple linear-Gaussian latent-variable model, the integral may have a closed form. But in a VAE, $p_\theta(x \mid z)$ is parameterized by a deep network $f_\theta(z)$. The latent space is typically continuous, often $\mathcal{Z} = \mathbb{R}^K$, and the integrand can be nonzero over an enormous region. We are therefore trying to integrate a nonlinear neural-network-shaped function over a high-dimensional space:

$$\int_{\mathbb{R}^K} p_\theta(x \mid z)\,p(z)\,dz.$$

There is no general symbolic simplification available.

A first instinct is to use Monte Carlo sampling from the prior. Draw

$$z^{(1)}, \dots, z^{(L)} \sim p(z),$$

approximate the expectation, and then take the logarithm:

$$\log p_{\theta}(x) = \log \mathbb{E}_{p(z)}\bigl[p_\theta(x \mid z)\bigr] \approx \log \frac{1}{L}\sum_{l=1}^{L} p_\theta(x \mid z^{(l)}).$$

This looks reasonable, and for very large $L$ it is a consistent estimator. But as a training objective, it has two serious issues. First, prior samples are usually a terrible way to find the latent codes that explain a specific datapoint $x$. In high dimensions, most $z \sim p(z)$ will decode to samples unrelated to $x$, so the average can be dominated by rare lucky samples. This leads to high variance.

Second, and more fundamentally, the log of a Monte Carlo average is a biased estimator of the log-evidence. The logarithm is concave, so Jensen’s inequality gives

$$\mathbb{E}\left[ \log \frac{1}{L}\sum_{l=1}^{L} p_\theta(x \mid z^{(l)}) \right] \leq \log \mathbb{E}\left[ \frac{1}{L}\sum_{l=1}^{L} p_\theta(x \mid z^{(l)}) \right] = \log p_\theta(x).$$

The inequality is strict unless the random quantity inside the log is essentially constant. Equivalently, in the single-sample case,

$$\log \mathbb{E}_{p(z)}\bigl[p_\theta(x \mid z)\bigr] \geq \mathbb{E}_{p(z)}\bigl[\log p_\theta(x \mid z)\bigr].$$

This is the key obstruction: moving the logarithm outside or inside an expectation changes the objective. The log-evidence is what we want, but it contains an intractable integral. The expectation of the log-likelihood is tractable by sampling, but it is generally a lower quantity and, by itself, is not the right maximum-likelihood objective.

This is where the VAE’s central idea begins to emerge. We need a surrogate objective that is:

• tractable by sampling, so it avoids exact integration over all $z$;
• a principled lower bound on $\log p_\theta(x)$, so optimizing it still pushes up the evidence;
• differentiable with low-variance gradients, so neural networks can be trained efficiently;
• aware of the datapoint $x$, so we sample useful latent codes rather than blind samples from the prior.

That surrogate will be the Evidence Lower Bound, or ELBO, written schematically as

$$\mathcal{L}(\theta,\phi; x) \leq \log p_\theta(x).$$

The additional parameter $\phi$ appears because we introduce an inference network $q_\phi(z \mid x)$, whose role is to propose latent codes likely to explain the observed datapoint. Instead of sampling $z$ blindly from the prior, the VAE learns where in latent space to look.

The visual below compresses this motivation into three layers: the maximum-likelihood goal, the intractable log-evidence integral, and the failed naive Monte Carlo shortcut. The red warning around Jensen’s inequality highlights the precise mathematical reason we cannot simply replace the integral with a sampled average inside a logarithm and proceed as if nothing changed.

The bottom layer then points to the resolution: rather than estimating the log-evidence directly, we construct a computable lower bound with good gradient behavior. The next step is to derive that bound carefully and see exactly why it is lower, when it becomes tight, and how it decomposes into the reconstruction and KL terms used in VAE training.

Having written the evidence as an integral over the latent variable, we now face the central obstacle of latent-variable maximum likelihood: the quantity we want,

$$\log p_{\theta}(x)=\log \int p_{\theta}(x,z)\,dz,$$

is usually not computable in closed form. The integral sums over all possible latent explanations $z$ of the observation $x$. For expressive neural decoders $p_{\theta}(x\mid z)$, this marginalization is precisely what makes the model powerful — and precisely what makes direct optimization difficult.

The key variational idea is to introduce an auxiliary distribution $q_{\phi}(z\mid x)$, which we will later implement as the encoder. At this point, however, $q_{\phi}$ is just a mathematical device: a distribution over latent variables conditioned on the observed datapoint. We use it to rewrite the intractable integral as an expectation under a distribution from which we can sample.

Starting from the evidence,

$$\log p_{\theta}(x) = \log \int p_{\theta}(x,z)\,dz,$$

we multiply and divide by $q_{\phi}(z\mid x)$:

$$\log p_{\theta}(x) = \log \int q_{\phi}(z\mid x) \frac{p_{\theta}(x,z)}{q_{\phi}(z\mid x)} \,dz.$$

This is an exact algebraic identity, not an approximation. It is the same logic as importance sampling: instead of integrating directly with respect to the latent variable measure, we express the integral as an expectation under a chosen proposal distribution $q_{\phi}(z\mid x)$:

$$\log p_{\theta}(x) = \log \mathbb{E}_{q_{\phi}(z\mid x)} \!\left[ \frac{p_{\theta}(x,z)}{q_{\phi}(z\mid x)} \right].$$

There is an important support condition hidden in this step. The ratio $p_{\theta}(x,z)/q_{\phi}(z\mid x)$ must be well-defined wherever $p_{\theta}(x,z)$ contributes mass. Informally, $q_{\phi}(z\mid x)$ must not assign zero probability to latent regions that the model considers possible explanations of $x$. Otherwise, the importance-weighted expression can become undefined or miss parts of the integral entirely.

Now comes the only inequality in the derivation. Since $\log$ is concave, Jensen’s inequality tells us that for a positive random variable $Y$,

$$\log \mathbb{E}[Y] \geq \mathbb{E}[\log Y].$$

Here the random variable is

$$Y = \frac{p_{\theta}(x,z)}{q_{\phi}(z\mid x)}, \qquad z\sim q_{\phi}(z\mid x).$$

Applying Jensen’s inequality gives

$$\log p_{\theta}(x) \geq \mathbb{E}_{q_{\phi}(z\mid x)} \left[ \log \frac{p_{\theta}(x,z)}{q_{\phi}(z\mid x)} \right],$$

or equivalently,

$$\log p_{\theta}(x) \geq \mathbb{E}_{q_{\phi}(z\mid x)} \!\left[ \log p_{\theta}(x,z) - \log q_{\phi}(z\mid x) \right].$$

This lower bound is the evidence lower bound, or ELBO:

$$\mathcal{L}(\theta,\phi;x) = \mathbb{E}_{q_{\phi}(z\mid x)} \!\left[ \log p_{\theta}(x,z) - \log q_{\phi}(z\mid x) \right].$$

The name is literal: it is a lower bound on the log-evidence $\log p_{\theta}(x)$. For any valid choice of $q_{\phi}(z\mid x)$,

$$\mathcal{L}(\theta,\phi;x) \leq \log p_{\theta}(x).$$

This is the foundational move in VAEs. We replace an intractable objective with a tractable lower bound that can be estimated by sampling from the encoder.

The bound becomes tight exactly when Jensen’s inequality becomes equality. For a concave function like $\log$, equality occurs when the random variable inside the expectation is constant almost surely under $q_{\phi}(z\mid x)$. In this case, we need

$$\frac{p_{\theta}(x,z)}{q_{\phi}(z\mid x)} = p_{\theta}(x) \quad \text{for } q_{\phi}\text{-almost every }z.$$

Rearranging,

$$q_{\phi}(z\mid x) = \frac{p_{\theta}(x,z)}{p_{\theta}(x)} = p_{\theta}(z\mid x).$$

So the ELBO is tight when the variational distribution $q_{\phi}(z\mid x)$ exactly matches the true posterior $p_{\theta}(z\mid x)$. This is also why the encoder is often described as an approximate posterior: it is trained to behave like the posterior distribution that would make the bound exact, but which is usually unavailable because it depends on the intractable evidence $p_{\theta}(x)$.

This derivation matters because it gives us a practical optimization target with a clear interpretation:

• maximizing the ELBO with respect to $\theta$ improves the generative model;
• maximizing it with respect to $\phi$ improves the approximate posterior;
• the gap between $\log p_{\theta}(x)$ and the ELBO measures how far $q_{\phi}(z\mid x)$ is from the true posterior.

There is a subtle but important caveat: maximizing a lower bound is not automatically the same as maximizing the original objective. If the bound is loose, we may improve $\mathcal{L}$ without significantly improving $\log p_{\theta}(x)$. The success of VAEs depends on making the variational family expressive enough, and the optimization stable enough, that this lower bound remains a useful surrogate for maximum likelihood.

The visual below condenses the derivation into its three essential moves: start from the marginal likelihood, rewrite it as an expectation using $q_{\phi}(z\mid x)$, then apply Jensen’s inequality to move the logarithm inside the expectation. The boxed expression is the ELBO, the quantity we can optimize in place of the inaccessible log-evidence.

It also highlights the geometric meaning of the inequality: $\mathcal{L}(\theta,\phi;x)$ sits below $\log p_{\theta}(x)$, with a gap determined by how imperfect the variational posterior is. In the next section, we will decompose this same ELBO into the two terms that make VAEs operational: a reconstruction term and a KL regularization term.

Having obtained the ELBO through Jensen’s inequality, the next question is: what exactly are we optimizing? The expression

$$\mathcal{L}(\theta, \phi; x) = \mathbb{E}_{q_{\phi}(z|x)} \!\left[ \log p_{\theta}(x,z) - \log q_{\phi}(z|x) \right]$$

is already a valid lower bound on $\log p_\theta(x)$, but in this form it hides the two competing forces that make VAEs work. To understand the training objective operationally, we want to rewrite it in terms of the decoder likelihood and a penalty on the encoder distribution.

The key move is simply to expand the joint model. In a VAE, the generative story is assumed to factor as: first sample a latent variable $z$ from a prior $p(z)$, then sample the observation $x$ from a decoder likelihood $p_\theta(x|z)$. Therefore,

$$p_\theta(x,z) = p_\theta(x|z)p(z),$$

and taking logs gives

$$\log p_\theta(x,z) = \log p_\theta(x|z) + \log p(z).$$

Substituting this into the ELBO gives

$$\mathcal{L}(\theta, \phi; x) = \mathbb{E}_{q_{\phi}(z|x)} \!\left[ \log p_\theta(x|z) + \log p(z) - \log q_\phi(z|x) \right].$$

Because expectation is linear, we can separate the terms:

$$\mathcal{L}(\theta, \phi; x) = \mathbb{E}_{q_{\phi}(z|x)} \!\left[ \log p_\theta(x|z) \right] + \mathbb{E}_{q_{\phi}(z|x)} \!\left[ \log p(z) - \log q_\phi(z|x) \right].$$

The first term has a direct modeling interpretation. We draw latent codes $z$ from the encoder distribution $q_\phi(z|x)$, then ask how much probability the decoder assigns back to the original observation $x$. This is the reconstruction term:

$$\mathbb{E}_{q_{\phi}(z|x)} \!\left[ \log p_\theta(x|z) \right].$$

It rewards latent representations that preserve information useful for explaining the input. If $p_\theta(x|z)$ is a Bernoulli distribution, this often corresponds to a binary cross-entropy reconstruction objective; if it is a Gaussian with fixed variance, it becomes proportional to a negative squared-error loss. This is why VAEs often look, at implementation time, like autoencoders with a probabilistic reconstruction loss.

The second term is more subtle. Recall that the KL divergence is

$$D_{\mathrm{KL}}\!\left(q_\phi(z|x)\,\|\,p(z)\right) = \mathbb{E}_{q_\phi(z|x)} \!\left[ \log q_\phi(z|x) - \log p(z) \right].$$

Our ELBO contains the negative of this quantity:

$$\mathbb{E}_{q_{\phi}(z|x)} \!\left[ \log p(z) - \log q_\phi(z|x) \right] = - D_{\mathrm{KL}}\!\left(q_\phi(z|x)\,\|\,p(z)\right).$$

So the ELBO decomposes into the now-famous form

$$\boxed{ \mathcal{L}(\theta, \phi; x) = \underbrace{ \mathbb{E}_{q_{\phi}(z|x)} \!\left[ \log p_\theta(x|z) \right] }_{\text{reconstruction term}} - \underbrace{ D_{\mathrm{KL}}\!\left(q_\phi(z|x)\,\|\,p(z)\right) }_{\text{regularisation term}} }$$

This decomposition explains the central tension in VAE training. The reconstruction term wants $q_\phi(z|x)$ to encode enough information about $x$ that the decoder can reconstruct it accurately. The KL term wants $q_\phi(z|x)$ to stay close to the prior $p(z)$, usually a simple standard Gaussian such as $\mathcal{N}(0,I)$. In other words, the encoder is not free to assign every datapoint an arbitrary isolated latent code; its codes must remain compatible with a shared latent space from which we can later sample.

This regularization is not merely aesthetic. Without it, the model could behave like an ordinary deterministic autoencoder: excellent reconstructions, but a latent space with holes, disconnected regions, and no reliable way to generate new samples by drawing $z \sim p(z)$. The KL penalty pushes the aggregate structure of the latent codes toward something smooth and sampleable. At the same time, if the KL pressure is too strong, the encoder may ignore $x$, producing $q_\phi(z|x) \approx p(z)$ for every input. That failure mode is known as posterior collapse, and it causes the latent variable to carry little or no information.

A useful way to remember the objective is:

• Reconstruction term: “Can the decoder explain this datapoint using latents from the encoder?”
• KL term: “Is the encoder’s posterior still close to the prior distribution we intend to sample from?”
• ELBO maximization: “Find a compromise between faithful reconstruction and a well-structured latent space.”

For the common Gaussian case, this tradeoff becomes especially convenient computationally. If

$$q_\phi(z|x) = \mathcal{N}(\mu_\phi(x), \operatorname{diag}(\sigma_\phi^2(x)))$$

and

$$p(z) = \mathcal{N}(0,I),$$

then the KL term has a closed-form expression. That means VAE training usually combines a Monte Carlo estimate of the reconstruction expectation with an analytic KL penalty, giving a stable and differentiable objective once we introduce the reparameterization trick.

The visual below condenses this derivation into its algebraic skeleton: start from the Jensen-derived ELBO, expand the joint distribution using the generative factorization, separate the expectation, and recognize the second piece as a negative KL divergence. The color split emphasizes that the final objective is not one monolithic loss, but a sum of two interpretable forces with opposite pressures.

Read the final boxed equation as the practical training objective for a single datapoint $x$: maximize expected decoder log-likelihood while minimizing deviation from the prior. This compact decomposition is the bridge between the abstract variational bound and the loss function we will soon implement in an actual VAE.

Having split the ELBO into a reconstruction term and a KL-to-prior regularizer, we can now ask a deeper question: what exactly did we lose when we replaced the true log-evidence $\log p_\theta(x)$ by the ELBO? The answer is one of the most important identities in variational inference: the missing quantity is not mysterious approximation error, nor a loose heuristic penalty. It is exactly a KL divergence between the variational encoder and the true Bayesian posterior.

For a latent-variable model, the evidence is

$$p_\theta(x)=\int p_\theta(x|z)p(z)\,dz,$$

and the true posterior over latents is

$$p_\theta(z|x) = \frac{p_\theta(x|z)p(z)}{p_\theta(x)}.$$

This posterior is the distribution we would ideally use for inference: after observing $x$, it tells us which latent explanations $z$ are plausible. The difficulty is that $p_\theta(z|x)$ depends on $p_\theta(x)$, the very integral we cannot usually compute. VAEs therefore introduce an encoder $q_\phi(z|x)$, a tractable approximation to this intractable posterior.

The central theorem says that, for a suitable variational distribution $q_\phi(z|x)$,

$$\log p_{\theta}(x) = \mathcal{L}(\theta,\phi;x) + D_{\mathrm{KL}} \!\left( q_{\phi}(z|x) \,\|\, p_{\theta}(z|x) \right).$$

Here the ELBO is

$$\mathcal{L}(\theta,\phi;x) = \mathbb{E}_{q_{\phi}(z|x)} \!\left[ \log p_{\theta}(x|z) \right] - D_{\mathrm{KL}} \!\left( q_{\phi}(z|x) \,\|\, p(z) \right).$$

So the ELBO is not merely inspired by the evidence. It is the evidence minus a very specific nonnegative gap.

Because KL divergence is always nonnegative,

$$D_{\mathrm{KL}} \!\left( q_{\phi}(z|x) \,\|\, p_{\theta}(z|x) \right) \geq 0,$$

we immediately recover the lower-bound property:

$$\mathcal{L}(\theta,\phi;x) \leq \log p_\theta(x).$$

This also explains the name Evidence Lower Bound. The ELBO sits below the log-evidence, and the vertical distance between them is precisely the mismatch between the approximate posterior $q_\phi(z|x)$ and the true posterior $p_\theta(z|x)$.

The theorem also tells us exactly when the bound is tight. We have

$$\mathcal{L}(\theta,\phi;x) = \log p_\theta(x)$$

if and only if

$$q_\phi(z|x)=p_\theta(z|x) \quad \text{almost everywhere}.$$

In words: the ELBO becomes the true log-evidence exactly when the encoder recovers the true posterior. There is no remaining variational gap. This is why the encoder is not just an auxiliary neural network used for amortized sampling; it is performing approximate Bayesian inference.

There is a subtle support assumption hiding here. For the KL

$$D_{\mathrm{KL}}\!\left(q_\phi(z|x)\,\|\,p_\theta(z|x)\right)$$

to be finite, $q_\phi$ should not place probability mass where the true posterior has zero mass. More formally, we usually require $q_\phi(z|x)$ to be absolutely continuous with respect to $p_\theta(z|x)$. At the same time, if the variational family is too restrictive and cannot represent the true posterior’s important regions, the gap cannot close. This is one reason why posterior approximation quality depends not only on optimization, but also on the expressiveness of the encoder family.

The identity has an important optimization consequence. Suppose $\theta$ is fixed. Then $\log p_\theta(x)$ is constant with respect to $\phi$, so maximizing the ELBO over $\phi$ is exactly equivalent to minimizing

$$D_{\mathrm{KL}} \!\left( q_\phi(z|x) \,\|\, p_\theta(z|x) \right).$$

That is the variational inference interpretation of the VAE encoder: training $q_\phi(z|x)$ by maximizing the ELBO pushes it toward the true posterior. When we also optimize $\theta$, we are doing two things at once: learning a generative model that assigns high probability to the data, and learning an approximate inference model that explains each datapoint in latent space.

A useful way to remember the theorem is:

• Evidence is the target quantity we wish we could maximize directly.
• ELBO is the tractable surrogate we can optimize.
• KL gap is the price paid for using $q_\phi(z|x)$ instead of the true posterior.

The visual below compresses this relationship into a geometric picture. The log-evidence is represented as a fixed point on a “nats” axis, while the ELBO lies to its left. The red interval between them is the posterior KL divergence. If the encoder improves, that interval shrinks; if $q_\phi(z|x)$ matches $p_\theta(z|x)$, the two points coincide.

The three corollaries in the visual are therefore not separate facts, but direct consequences of the same decomposition: the ELBO is a lower bound because KL is nonnegative; it is tight exactly when the approximate and true posteriors agree; and maximizing it with respect to $\phi$ is variational inference. The next step is to prove the identity algebraically, showing how Bayes’ rule turns the apparently inaccessible evidence into the sum of an optimizable lower bound and an explicit posterior mismatch term.

Having stated that the gap between the log evidence and the ELBO is a KL divergence, we should now verify that this is not a heuristic slogan. It is an exact identity. The derivation is short, but it is worth reading carefully because it explains why variational inference works: we are not inventing an arbitrary surrogate objective; we are decomposing the true marginal log-likelihood into a tractable lower bound plus a nonnegative error term.

Fix one observed datapoint $x$. The model defines a latent-variable joint distribution

$$p_{\theta}(x,z) = p_{\theta}(x|z)p(z),$$

and the true posterior is

$$p_{\theta}(z|x) = \frac{p_{\theta}(x,z)}{p_{\theta}(x)}.$$

The problem is that $p_{\theta}(x)$, the evidence, usually requires integrating out $z$:

$$p_{\theta}(x) = \int p_{\theta}(x,z)\,dz.$$

In a VAE, this integral is generally intractable because the decoder $p_{\theta}(x|z)$ is represented by a neural network. So instead of working directly with the true posterior $p_{\theta}(z|x)$, we introduce an approximate posterior, or encoder distribution,

$$q_{\phi}(z|x).$$

The key question is: how does optimizing an objective involving $q_{\phi}(z|x)$ relate to maximizing the true log evidence $\log p_{\theta}(x)$?

Start from the KL divergence between the variational posterior and the true posterior:

$$D_{\mathrm{KL}}\!\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z|x)\right) = \mathbb{E}_{q_{\phi}(z|x)} \left[ \log q_{\phi}(z|x) - \log p_{\theta}(z|x) \right].$$

Now substitute Bayes’ rule into the true posterior term:

$$p_{\theta}(z|x) = \frac{p_{\theta}(x,z)}{p_{\theta}(x)}.$$

Taking logs gives

$$\log p_{\theta}(z|x) = \log p_{\theta}(x,z) - \log p_{\theta}(x).$$

Therefore,

$$D_{\mathrm{KL}}\!\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z|x)\right) = \mathbb{E}_{q_{\phi}(z|x)} \left[ \log q_{\phi}(z|x) - \log p_{\theta}(x,z) + \log p_{\theta}(x) \right].$$

The subtle but crucial observation is that $\log p_{\theta}(x)$ does not depend on $z$. The expectation is over $z \sim q_{\phi}(z|x)$, while $x$ is fixed. So we can pull $\log p_{\theta}(x)$ outside the expectation:

$$D_{\mathrm{KL}}\!\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z|x)\right) = \mathbb{E}_{q_{\phi}(z|x)} \left[ \log q_{\phi}(z|x) - \log p_{\theta}(x,z) \right] + \log p_{\theta}(x).$$

Equivalently,

$$D_{\mathrm{KL}}\!\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z|x)\right) = - \mathbb{E}_{q_{\phi}(z|x)} \left[ \log p_{\theta}(x,z) - \log q_{\phi}(z|x) \right] + \log p_{\theta}(x).$$

The expectation inside the negative sign is precisely the evidence lower bound:

$$\mathcal{L}(\theta,\phi;x) = \mathbb{E}_{q_{\phi}(z|x)} \left[ \log p_{\theta}(x,z) - \log q_{\phi}(z|x) \right].$$

So the KL divergence becomes

$$D_{\mathrm{KL}}\!\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z|x)\right) = -\mathcal{L}(\theta,\phi;x) + \log p_{\theta}(x).$$

Rearranging gives the central decomposition:

$$\boxed{ \log p_{\theta}(x) = \mathcal{L}(\theta,\phi;x) + D_{\mathrm{KL}}\!\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z|x)\right) }$$

This is the whole proof. No approximation has been made. We used only the definition of KL divergence, Bayes’ theorem, and the fact that constants can be moved outside expectations.

The consequence is immediate. Since KL divergence is always nonnegative,

$$D_{\mathrm{KL}}\!\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z|x)\right) \geq 0,$$

we have

$$\log p_{\theta}(x) \geq \mathcal{L}(\theta,\phi;x).$$

That is why $\mathcal{L}$ is called a lower bound on the log evidence. The bound is tight exactly when

$$D_{\mathrm{KL}}\!\left(q_{\phi}(z|x)\,\|\,p_{\theta}(z|x)\right) = 0,$$

which occurs when

$$q_{\phi}(z|x) = p_{\theta}(z|x)$$

almost everywhere under $q_{\phi}$. In words: the ELBO equals the true log evidence when the encoder distribution exactly matches the model’s true posterior.

This identity also clarifies a common failure mode in variational modeling. Maximizing the ELBO improves a lower bound on $\log p_{\theta}(x)$, but it does so through the restricted family $q_{\phi}(z|x)$. If the encoder family is too limited, the KL gap may remain large even after optimization. The objective is still valid, but the approximation may be poor. Conversely, if $q_{\phi}$ is expressive enough and optimization succeeds, the ELBO can become a very accurate proxy for the true marginal likelihood.

The visual below compresses the derivation into a chain of equalities: start with the KL divergence to the true posterior, replace the posterior using Bayes’ theorem, separate the constant $\log p_{\theta}(x)$, recognize the ELBO, and rearrange. The boxed final identity is not an additional assumption; it is simply the same expression written so that the evidence appears on the left.

It is useful to keep this picture in mind as we move forward. The ELBO is not merely “reconstruction plus regularization” yet—that interpretation will come after expanding the joint model. At this stage, the most important fact is structural: log evidence decomposes exactly into an optimizable lower bound plus a nonnegative posterior-approximation gap.

Having just seen that the ELBO differs from the true log evidence by a nonnegative KL gap, we can now reinterpret the bound in a more operational way. The ELBO is not merely an algebraic trick for lower-bounding $\log p_\theta(x)$; it is the objective that tells a VAE how to trade off using latent information against staying compatible with the prior.

For a single datapoint $x$, the VAE objective is

$$\mathcal{L}(\theta, \phi; x) = \mathbb{E}_{q_{\phi}(z|x)}[\log p_{\theta}(x|z)] - D_{\mathrm{KL}}(q_{\phi}(z|x) \| p(z)).$$

The first term is the reconstruction term. It asks: if the encoder distribution $q_\phi(z|x)$ samples a latent code $z$, how much probability does the decoder $p_\theta(x|z)$ assign back to the original observation $x$? Maximizing this term encourages the latent representation to preserve information that the decoder needs. In an image VAE, this means encoding shape, pose, color, texture, or any features that help predict the input under the chosen likelihood model.

The second term is the regularization term, but that phrase can be slightly misleading if we think of it as a generic penalty. More precisely, it penalizes the encoder distribution $q_\phi(z|x)$ for moving too far away from the prior $p(z)$, usually $p(z)=\mathcal{N}(0,I)$. This matters because generation will later sample $z \sim p(z)$, not $z \sim q_\phi(z|x)$. If the encoder learns latent codes that live in strange isolated regions far from the prior, then samples drawn from the prior may decode poorly. The KL term forces the aggregate geometry of the latent space to remain usable for generation.

We can unpack the KL penalty further using

$$D_{\mathrm{KL}}(q\|p) = -\mathcal{H}[q] + \mathbb{E}_{q}[-\log p(z)].$$

Therefore,

$$-D_{\mathrm{KL}}(q_{\phi}(z|x)\|p(z)) = \mathcal{H}[q_{\phi}(z|x)] - \mathbb{E}_{q_{\phi}(z|x)}[-\log p(z)].$$

This decomposition reveals two forces hidden inside the regularization term. The entropy term $\mathcal{H}[q_\phi(z|x)]$ rewards the encoder for being uncertain, or spread out, rather than collapsing to a deterministic point code. Meanwhile, the expected negative log prior term penalizes codes that fall in regions where $p(z)$ assigns low probability. For a standard Gaussian prior, this means the model is discouraged from placing latent mass far away from the origin.

So the ELBO is balancing two competing desires:

• Reconstruction: encode enough information in $z$ to explain $x$ well.
• Regularization: keep $q_\phi(z|x)$ close enough to $p(z)$ that latent samples remain meaningful.

This tension is fundamental. If the KL penalty is too weak, the encoder may memorize examples using highly specialized latent codes, producing good reconstructions but a poorly organized latent space. If the KL penalty is too strong, the encoder may ignore $x$ and make $q_\phi(z|x)\approx p(z)$, which can lead to posterior collapse: the decoder receives little useful information from $z$, and the latent variables stop carrying semantic structure.

This is why the ELBO also has a natural rate–distortion interpretation. The KL term acts like a rate: it measures how many nats, or bits up to a constant conversion, are required to encode $z$ using a distribution different from the prior. The reconstruction error acts like a distortion: poor reconstructions correspond to high distortion, while high log-likelihood corresponds to low distortion. Maximizing the ELBO means finding a good operating point between compression cost and reconstruction fidelity.

The connection to physics and variational inference comes from writing the training problem as minimization of the negative ELBO:

$$-\mathcal{L}(\theta,\phi;x).$$

This quantity is often called the variational free energy or negative evidence lower bound. Since we previously established

$$\log p_{\theta}(x) = \mathcal{L}(\theta, \phi; x) + D_{\mathrm{KL}}(q_{\phi}(z|x)\|p_{\theta}(z|x)),$$

minimizing free energy simultaneously pushes the ELBO upward and tries to reduce the gap between the approximate posterior $q_\phi(z|x)$ and the true posterior $p_\theta(z|x)$. The bound becomes tight exactly when

$$q_\phi(z|x)=p_\theta(z|x),$$

assuming the variational family is expressive enough to represent the true posterior.

The visual below condenses this interpretation into a balance: reconstruction pulls the objective toward faithful recovery of $x$, while KL regularization pulls the encoder distribution back toward the prior. The scale metaphor is useful because neither side is “bad”; a good VAE needs both. Reconstruction gives the latent variable meaning, while the KL term gives the latent space global structure.

It also highlights the rate–distortion view: moving toward lower distortion usually requires spending more rate, while enforcing a very low rate often sacrifices detail. Much of practical VAE design—choosing decoder likelihoods, KL schedules, $\beta$-VAE objectives, or more expressive priors—is about controlling exactly where this operating point lies.

Having seen the ELBO as a negative free energy, it is worth pausing on a case where nothing is hidden behind approximation. In most VAE settings, the posterior $p_\theta(z \mid x)$ is intractable, which is exactly why we introduce a variational distribution $q_\phi(z \mid x)$. But in a one-dimensional linear-Gaussian model, the posterior, marginal likelihood, KL terms, and ELBO can all be written down exactly. That makes it a useful sanity check: if our interpretation of the ELBO is right, then optimizing it over a sufficiently expressive variational family should recover the true posterior and make the ELBO equal to the evidence.

Consider the toy generative model

$$p(z) = \mathcal{N}(0,1), \qquad p_\theta(x \mid z) = \mathcal{N}(z, 0.1).$$

Here the latent variable $z$ is drawn from a standard normal prior, and the observation $x$ is a noisy version of $z$ with small Gaussian noise variance $0.1$. Because both the prior and likelihood are Gaussian, the marginal distribution of $x$ is also Gaussian:

$$p_\theta(x) = \mathcal{N}(0, 1.1),$$

so the log evidence is available in closed form:

$$\log p_\theta(x) = -\frac{1}{2}\log(2\pi \cdot 1.1) - \frac{x^2}{2.2}.$$

The true posterior is also Gaussian. Combining the prior precision $1$ with the likelihood precision $1/0.1 = 10$, the posterior variance is

$$(\sigma^*)^2 = \frac{1}{1 + 10} = \frac{0.1}{1.1},$$

and the posterior mean is the precision-weighted estimate

$$\mu^* = \frac{x}{1.1}.$$

Thus

$$p_\theta(z \mid x) = \mathcal{N}\!\left(\mu^*,(\sigma^*)^2\right), \qquad \mu^* = \frac{x}{1.1}, \qquad (\sigma^*)^2 = \frac{0.1}{1.1}.$$

This already contains an important intuition. The observation $x$ pulls the posterior mean toward $x$, but not all the way: the prior still shrinks the estimate toward zero. Since the observation noise is small, the posterior variance is much smaller than the prior variance. For $x=1$, we get

$$\mu^* \approx 0.91, \qquad \sigma^* \approx 0.30.$$

Now suppose our variational family is

$$q_\phi(z \mid x) = \mathcal{N}(\mu,\sigma^2).$$

This family is expressive enough to contain the true posterior exactly, because the true posterior is also a one-dimensional Gaussian. The ELBO is

$$\mathcal{L}(\theta,\phi;x) = \mathbb{E}_{q_\phi(z \mid x)}[\log p_\theta(x \mid z)] - D_{\mathrm{KL}}(q_\phi(z \mid x)\,\|\,p(z)).$$

In this Gaussian case, both terms can be evaluated analytically. The expected reconstruction term is

$$\mathbb{E}_{q_\phi}[\log p_\theta(x \mid z)] = -\frac{1}{2}\log(2\pi \cdot 0.1) - \frac{(x-\mu)^2+\sigma^2}{2\cdot 0.1},$$

because under $q_\phi$,

$$\mathbb{E}_{q_\phi}[(x-z)^2] = (x-\mu)^2+\sigma^2.$$

The KL regularizer against the standard normal prior is

$$D_{\mathrm{KL}}\!\left(\mathcal{N}(\mu,\sigma^2)\,\|\,\mathcal{N}(0,1)\right) = \frac{1}{2}\left(\mu^2+\sigma^2-1-\log\sigma^2\right).$$

So the ELBO is a deterministic function of only two variational parameters, $\mu$ and $\sigma$. There is no Monte Carlo noise, no neural network approximation, and no optimization mystery. We can inspect the entire objective surface directly.

The key identity from the previous section was

$$\log p_\theta(x) = \mathcal{L}(\theta,\phi;x) + D_{\mathrm{KL}}(q_\phi(z \mid x)\,\|\,p_\theta(z \mid x)).$$

Because the evidence $\log p_\theta(x)$ does not depend on $\phi$, maximizing the ELBO over $\phi$ is equivalent to minimizing

$$D_{\mathrm{KL}}(q_\phi(z \mid x)\,\|\,p_\theta(z \mid x)).$$

In this toy example, the minimum possible KL gap is exactly zero, since the variational family contains the true posterior. Therefore the global maximizer is

$$q_\phi(z \mid x) = p_\theta(z \mid x),$$

or equivalently,

$$\mu = \mu^*, \qquad \sigma = \sigma^*.$$

At that point,

$$D_{\mathrm{KL}}(q_\phi(z \mid x)\,\|\,p_\theta(z \mid x)) = 0,$$

and the ELBO becomes tight:

$$\mathcal{L}(\theta,\phi;x) = \log p_\theta(x).$$

This is the tightness corollary in its cleanest possible form.

It is useful to compare three operating points for $x=1$:

• Prior proposal: $q(z \mid x)=\mathcal{N}(0,1)$. This ignores the observation entirely. It has no KL cost relative to the prior, but it reconstructs $x=1$ poorly, so the ELBO is low.
• Centered but too wide: $q(z \mid x)=\mathcal{N}(\mu^*, (3\sigma^*)^2)$. This puts its mean in the right place but spreads too much probability mass over implausible latent values. The ELBO improves, but the KL gap to the true posterior remains positive.
• Exact posterior: $q(z \mid x)=\mathcal{N}(\mu^*,(\sigma^*)^2)$. This matches both the mean and uncertainty of the true posterior, so the KL gap vanishes and the ELBO reaches $\log p_\theta(x)$.

The visual below consolidates these facts in two complementary ways. The contour plot treats the ELBO as a surface over $(\mu,\sigma)$: the maximum occurs exactly at the true posterior parameters, not merely near them. The prior-like approximation sits far away from the optimum, while the centered-but-wide approximation lands on the correct mean but remains suboptimal because its uncertainty is wrong.

The companion bar chart emphasizes the decomposition

$$\log p_\theta(x) = \mathcal{L}(\theta,\phi;x) + D_{\mathrm{KL}}(q_\phi(z \mid x)\,\|\,p_\theta(z \mid x)).$$

For each candidate $q_\phi$, the ELBO plus the KL gap reaches the same evidence level. In the exact posterior case, the red “gap” disappears entirely, making the lower bound not just a bound, but the true log marginal likelihood.

The one-dimensional Gaussian example makes the ELBO feel almost deceptively friendly: the KL term can be written down, the expectation can sometimes be evaluated or estimated, and the whole objective looks like something we should be able to optimize with ordinary backpropagation. But the moment we replace that toy setup with a neural encoder and decoder, a subtle obstruction appears. The problem is not that the ELBO is undefined, nor that Monte Carlo estimation is impossible. The problem is that the Monte Carlo samples themselves sit in the middle of the computation graph.

Recall that for a single datapoint $x$, the VAE objective is

$$\mathcal{L}(\theta,\phi;x) = \mathbb{E}_{q_{\phi}(z|x)} \left[ \log p_{\theta}(x|z) \right] - D_{\mathrm{KL}}\bigl(q_{\phi}(z|x)\|p(z)\bigr).$$

When we differentiate with respect to the encoder parameters $\phi$, the gradient decomposes as

$$\nabla_{\phi}\,\mathcal{L}(\theta,\phi;x) = \underbrace{ \nabla_{\phi}\, \mathbb{E}_{q_{\phi}(z|x)} \left[ \log p_{\theta}(x|z) \right] }_{\text{reconstruction term: hard}} - \underbrace{ \nabla_{\phi}\, D_{\mathrm{KL}}\bigl(q_{\phi}(z|x)\|p(z)\bigr) }_{\text{regularization term: usually easy}}.$$

The KL term is usually not the source of trouble in the standard VAE. If $q_{\phi}(z|x)$ is a diagonal Gaussian,

$$q_{\phi}(z|x)=\mathcal{N}\bigl(z;\mu_{\phi}(x), \operatorname{diag}(\sigma_{\phi}^2(x))\bigr),$$

and the prior is $p(z)=\mathcal{N}(0,I)$, then the KL divergence has a closed form. It is just a differentiable expression involving $\mu_{\phi}(x)$ and $\sigma_{\phi}(x)$. Autodiff can handle this directly.

The reconstruction term is different because the distribution inside the expectation depends on $\phi$:

$$\mathbb{E}_{q_{\phi}(z|x)} \left[ \log p_{\theta}(x|z) \right] = \int q_{\phi}(z|x)\, \log p_{\theta}(x|z)\,dz.$$

Here $\phi$ controls where we sample from. Changing $\phi$ changes the density $q_{\phi}(z|x)$, which changes the regions of latent space that contribute to the expectation. This is not the same as differentiating an ordinary deterministic function. The parameter $\phi$ affects the objective through the sampling distribution itself.

A natural first attempt is to use Monte Carlo sampling. Draw

$$z^{(l)} \sim q_{\phi}(z|x),$$

and approximate the reconstruction expectation by

$$\tilde{\mathcal{L}} = \frac{1}{L} \sum_{l=1}^{L} \log p_{\theta}(x|z^{(l)}).$$

This gives an unbiased estimate of the expectation. For optimizing $\theta$, it is fine: once $z^{(l)}$ is sampled, the decoder likelihood $\log p_{\theta}(x|z^{(l)})$ is an ordinary differentiable function of $\theta$. But for optimizing $\phi$, naive backpropagation runs into a wall. The sampled value $z^{(l)}$ is not represented as a differentiable deterministic function of $\phi$. In the computation graph, the operation “sample from $q_{\phi}(z|x)$” behaves like a stochastic node, not like a smooth layer.

This is the key failure mode of naive Monte Carlo:

$$\nabla_{\phi} \left[ \frac{1}{L} \sum_{l=1}^{L} \log p_{\theta}(x|z^{(l)}) \right]$$

does not correctly estimate

$$\nabla_{\phi} \mathbb{E}_{q_{\phi}(z|x)} \left[ \log p_{\theta}(x|z) \right].$$

If we treat the sampled $z^{(l)}$ values as fixed constants after sampling, then the reconstruction term has no differentiable path back to $\phi$. Informally, the gradient is blocked at the sampling operation. More precisely, the particular sampled numerical value may change if we rerun the sampler with a different $\phi$, but that dependence is not available to standard backpropagation as a local derivative through the realized sample.

There is a general-purpose workaround called the score-function estimator, often known in this context as REINFORCE. It uses the identity

$$\nabla_{\phi}\, \mathbb{E}_{q_{\phi}}[f(z)] = \mathbb{E}_{q_{\phi}} \left[ f(z)\,\nabla_{\phi}\log q_{\phi}(z|x) \right].$$

This identity is valid under mild regularity assumptions and does not require differentiating through the sampled value $z$. Instead, it differentiates the log-density of the sampling distribution. That is powerful: it works even for discrete random variables, where ordinary pathwise derivatives are unavailable.

But for VAEs with neural decoders, this estimator is usually too noisy to be practical without substantial variance reduction. The scalar reward-like term $f(z)=\log p_{\theta}(x|z)$ can vary dramatically across samples, especially early in training. Multiplying it by $\nabla_{\phi}\log q_{\phi}(z|x)$ often produces gradients with very high variance. In principle the estimator is unbiased; in practice, its updates can be so unstable that learning becomes slow, erratic, or ineffective.

So the issue is not merely “sampling is random.” The deeper issue is that we need a low-variance differentiable estimator of how the reconstruction objective changes when the encoder distribution changes. Naive Monte Carlo gives samples but no usable pathwise gradient. REINFORCE gives a gradient but often with prohibitive variance. This tension is exactly what motivates the reparameterization trick.

The visual below compresses this obstruction into a computation graph. The forward pass is straightforward: $x$ is encoded into parameters of $q_{\phi}(z|x)$, a latent sample $z^{(l)}$ is drawn, and the decoder evaluates $\log p_{\theta}(x|z^{(l)})$. The backward pass is where the asymmetry appears: gradients flow cleanly through the decoder, but the attempted gradient path back through the stochastic sampling node is blocked.

The small REINFORCE annotation is the important caveat. There is a mathematical gradient estimator that avoids differentiating through the sample, but it pays for that generality with high variance. The next step is to replace the problematic sampling operation with an equivalent differentiable construction—one that preserves the same distribution over $z$, while allowing backpropagation to reach $\phi$.