We now want a failure case that is simple enough to reason about in one glance, but rich enough to expose the real limitation of value-based control. The key idea is perceptual aliasing: the environment may be fully Markov in the underlying state, yet the agent’s observation collapses several distinct states into the same feature vector. Once that happens, the policy no longer gets to condition on the true state, only on what it can see.

That distinction matters because a value method such as Q-learning ultimately extracts a deterministic control rule, $$\pi^*(s)=\arg\max_a Q^\pi(s,a),$$ so every observation is mapped to a single action. If two or more hidden states share the same observation but require different actions, then a deterministic policy is forced to “average over” incompatible choices by committing to one of them. In a standard MDP with full observability, that is usually fine; in an aliased setting, it can be fatal.

A small gridworld makes the issue concrete. Imagine a $5\times 5$ grid with a goal in one corner and four interior squares that all look identical locally: each has walls on the same two sides, so the agent sees the same feature vector in all four places. The true state differs, but the observation does not. From the agent’s perspective, these are not four distinct situations — they are one ambiguous observation repeated in four locations.

Now ask what a deterministic policy can do. Because it must output a single action for the shared observation, it must choose the same move in each aliased square. But no single move is simultaneously correct everywhere: an action that escapes one hidden configuration may hit a wall or send the agent into a loop in another. In this kind of construction, any fixed choice incurs a substantial failure probability, and the expected return stays low. The problem is not that the policy is poorly trained; the problem is that the policy class is too rigid.

A stochastic policy is different in exactly the right way. Instead of collapsing the decision to one action, it can represent a distribution $$\pi_\theta(a\mid s)=0.5 \quad \text{for } a\in\{\text{N},\text{E}\},$$ or symmetrically over $\{\text{S},\text{W}\}$, so that the agent randomizes between two perpendicular escape directions. This is not indecision for its own sake; it is a principled response to partial observability. When the same observation corresponds to several hidden states, randomization can hedge against the hidden ambiguity and produce a much higher return than any deterministic commitment.

This is exactly why the usual greedy value-based view can break down here. The maximizer $\arg\max_a Q^\pi(s,a)$ assumes that one action should dominate for the state under consideration. But when the observation aliases multiple hidden states, the induced greedy policy is still deterministic and therefore cannot adapt to each hidden case separately. Even if the Q-function is accurate with respect to the underlying hidden dynamics, the final policy extraction step throws away the very stochasticity that would solve the task.

The broader lesson is that we should optimize the policy directly, not merely infer it indirectly from values. By parameterizing $\pi_\theta(a\mid s)$ and maximizing $J(\theta)$, we allow the model class itself to include useful stochastic strategies. That is the real motivation for policy gradients: they are not just another way to train controllers, but a way to learn policies whose distributional structure is essential to success.

The visual below compresses this argument into two contrasted worlds. On the left, the deterministic controller is trapped by a single committed action: the same red arrow must serve every aliased square, so at least some hidden cases fail or cycle. On the right, the stochastic policy assigns probability mass to multiple actions, and that extra flexibility lets the agent escape the ambiguous region and reach the goal with high probability.

If you keep one idea from this example, let it be this: when the observation is ambiguous, the right object to learn is not just a value function, but a parameterized policy distribution. This is the conceptual bridge from “why not just learn a value function?” to the policy-gradient objective that follows.

After the failure modes of aliasing, the next question is almost unavoidable: what exactly are we optimizing when we say “learn a policy”? In policy gradients, the answer is not a value table or a greedy action rule, but a parameterized stochastic policy $\pi_\theta$ whose parameters $\theta \in \mathbb{R}^d$ can be adjusted by gradient ascent. Formally, at each time $t$, the policy defines a distribution over actions, $$\pi_\theta(a \mid s) = P(A_t = a \mid S_t = s;\, \theta),$$ so the policy is not a single action choice but a family of action probabilities that can be made sharper, smoother, or more exploratory depending on $\theta$.

The objective in episodic reinforcement learning is to choose $\theta$ so that trajectories with high long-term return become more likely. A standard way to write this is $$J(\theta) = \mathbb{E}_{\pi_\theta}[G_0],$$ where $G_0$ is the discounted return from the start of an episode. This is already useful conceptually: instead of optimizing immediate reward, we optimize the expected total outcome under the policy’s own behavior. If the initial state is random, we can also view this as an average over the initial-state distribution $\mu(s)$: $$J(\theta) = \sum_{s \in \mathcal{S}} \mu(s)\, V^{\pi_\theta}(s).$$ This makes the dependence on the policy explicit through the value function $V^{\pi_\theta}$, while separating it from the environment’s starting-state distribution.

It is often cleaner to reason not at the level of states, but at the level of entire trajectories. A trajectory $\tau = (s_0,a_0,s_1,a_1,\dots,s_T)$ is a full episode sampled from the policy and the environment dynamics, and the same objective can be written as $$J(\theta) = \mathbb{E}_{\tau}\bigl[R(\tau)\bigr], \qquad R(\tau) = \sum_{t=0}^{T} \gamma^t R_{t+1}.$$ This trajectory view is the one that later makes the likelihood-ratio trick work. It turns the objective into an expectation over a distribution $p(\tau \mid \theta)$, which is the key object we need when differentiating $J(\theta)$.

The crucial structural fact is that the trajectory probability factorizes into two very different parts: $$p(\tau \mid \theta) = \mu(s_0)\prod_{t=0}^{T-1} P(s_{t+1} \mid s_t, a_t)\,\pi_\theta(a_t \mid s_t).$$ Only the policy terms depend on $\theta$. The environment transition probabilities $P(s_{t+1}\mid s_t,a_t)$ and the initial-state distribution $\mu(s_0)$ are fixed by the environment. This distinction matters enormously: it is what makes policy gradients possible even when the dynamics are unknown, non-differentiable, or too complex to model directly. We never need to differentiate through the environment; we only need to know how the policy assigns probability to the actions it takes.

This also explains why policy-gradient methods are so different from value-based methods. Instead of solving for a greedy argmax over actions, we directly reshape the action distribution so that trajectories with larger return become more likely. In that sense, the optimization problem is:

• given a family of stochastic policies $\pi_\theta$,
• measure performance by expected discounted return $J(\theta)$,
• update parameters by ascent, $$\theta \leftarrow \theta + \alpha\, \nabla_\theta J(\theta).$$

That update rule is deceptively simple, because the real challenge is estimating $\nabla_\theta J(\theta)$ from sampled experience. But once the objective has been written at the trajectory level, the path to REINFORCE and the policy gradient theorem becomes natural: we will differentiate an expectation over trajectories, isolate the $\theta$-dependent policy factors, and turn return-weighted action probabilities into a usable learning signal.

The visual below compresses exactly that logic. The left side collects the same objective in three equivalent forms: policy, state-value, and trajectory expectation. That progression is not redundant; it shows that we can move between local decisions and global episode performance without changing the underlying quantity we optimize. The final boxed update rule at the bottom is there to remind you that all of this structure ultimately feeds one operation: gradient ascent on $J(\theta)$.

On the right, the unrolled trajectory diagram reinforces the most important modeling asymmetry: the blue arrows are controlled by $\theta$, while the gray arrows belong to the environment. That is the conceptual hinge of policy gradients. We are not trying to redesign the world; we are only adjusting how the agent samples actions inside it.

After introducing the policy objective, the next question is not why we would optimize a stochastic policy, but how we can take gradients of it in a form that is actually usable. The key object is the score function $\nabla_\theta \log \pi_\theta(a \mid s)$: once this quantity is available, the policy gradient machinery becomes concrete, because the gradient of the return can be written as an expectation of that score weighted by advantage-like signals.

The point of this section is that two standard policy classes give us exactly the kind of differentiable structure we need. They look different—one for discrete action spaces, one for continuous action spaces—but both produce a clean log-derivative that can be computed by ordinary backpropagation. That is the crucial bridge between abstract policy optimization and implementable algorithms.

For a discrete action set, the most common choice is the softmax policy. If we assign each action $a$ a feature vector $\phi(s,a)$, then the policy is $$\pi_\theta(a \mid s) = \frac{\exp(\phi(s,a)^\top \theta)}{\sum_{a'\in\mathcal{A}} \exp(\phi(s,a')^\top \theta)}.$$ This is just a normalized exponential family model over actions: the numerator prefers actions whose features align with $\theta$, and the denominator ensures the probabilities sum to one. The shape of the resulting gradient is especially elegant: $$\nabla_\theta \log \pi_\theta(a \mid s) = \phi(s,a) - \sum_{a'\in\mathcal{A}} \pi_\theta(a'\mid s)\,\phi(s,a').$$ In words, the update direction is the observed feature vector minus the policy’s own expected feature vector. That centering matters: it prevents the gradient from simply chasing large feature magnitudes and instead measures how surprising the chosen action was relative to the current policy. This is one reason softmax policies behave so naturally in REINFORCE-style updates.

For continuous actions, the canonical choice is a Gaussian policy. Instead of selecting an action from a finite set, the policy outputs a mean $\mu(s;\theta)$ and samples around it: $$\pi_\theta(a \mid s)=\frac{1}{\sigma\sqrt{2\pi}}\exp\!\left(-\frac{(a-\mu(s;\theta))^2}{2\sigma^2}\right).$$ Here the policy class is smooth in the parameters through the mean network $\mu(s;\theta)$, and the score function becomes $$\nabla_\theta \log \pi_\theta(a \mid s) = \frac{a-\mu(s;\theta)}{\sigma^2}\,\nabla_\theta \mu(s;\theta).$$ This expression is extremely informative. The scalar factor $(a-\mu)/\sigma^2$ says that actions sampled above the mean and below the mean push the network in opposite directions, with larger deviations creating stronger learning signals. The vector factor $\nabla_\theta \mu(s;\theta)$ is where standard backpropagation enters: once the mean is represented by a neural network, the policy gradient can be propagated through it exactly like any other differentiable model.

A useful way to compare the two cases is:

• Softmax: directly parameterizes a categorical distribution over actions.
• Gaussian: parameterizes a density over a real-valued action.
• Both: yield a closed-form score function $\nabla_\theta \log \pi_\theta(a\mid s)$, which is the quantity needed by policy gradient methods.
• Both: are smooth in $\theta$, so the optimization can proceed by standard gradient ascent under the usual regularity assumptions.

This smoothness is not a cosmetic detail. Policy gradient theory relies on being able to exchange differentiation and expectation, so the parameterization must be well-behaved enough for the gradient of $J(\theta)$ to pass through the stochastic action selection. Softmax and Gaussian policies are the workhorses because they satisfy this requirement while still remaining simple enough to evaluate and differentiate exactly.

The visual below condenses that story into two parallel pipelines. The left panel captures the softmax case as “scores $\rightarrow$ probabilities $\rightarrow$ centered features,” while the right panel shows the Gaussian case as “mean network $\rightarrow$ density $\rightarrow$ backpropagated score.” Read together, they reinforce the main lesson: the policy gradient method does not need to differentiate through the sampling operation itself; it only needs a policy class whose log-derivative is tractable.

That is why these two parameterizations appear so early in the policy-gradient story. They are not merely convenient examples; they are the concrete templates from which REINFORCE, baseline methods, and actor-critic algorithms all inherit their gradient estimates.

Having defined a policy class, the next question is not what to optimize, but how to differentiate the objective efficiently. In policy gradient methods, we want to maximize

$$J(\theta)=\mathbb{E}_{\tau \sim p(\tau\mid \theta)}[R(\tau)],$$

where $\tau$ denotes an entire trajectory and $R(\tau)$ is the return accumulated along it. The key subtlety is that $\theta$ does not appear inside the reward function itself; it appears in the distribution over trajectories. That means the objective is an expectation whose measure depends on the parameters we are trying to optimize. This is exactly the kind of situation where naive differentiation becomes awkward.

If we write the gradient directly, we get

$$\nabla_\theta J(\theta)=\nabla_\theta \int R(\tau)\,p(\tau\mid \theta)\,d\tau.$$

At first glance, this looks like something we might pass through the integral sign. But that instinct hides the real difficulty: the integrand is not just $R(\tau)$, it is $R(\tau)$ times a parameter-dependent density. The domain of integration over trajectories does not move, but the probability mass assigned to each trajectory does. So the challenge is not “differentiate the reward”; it is “differentiate through the sampling distribution that generates the data.”

This distinction matters because in reinforcement learning, the return $R(\tau)$ is often noisy, delayed, and discontinuous with respect to actions. There is usually no differentiable computational graph from $\theta$ to $R(\tau)$ in the way that standard backpropagation assumes. Instead, the policy influences the likelihood of seeing different trajectories:

$$p(\tau\mid \theta)=\mu(s_0)\prod_{t=0}^{T-1} P(s_{t+1}\mid s_t,a_t)\,\pi_\theta(a_t\mid s_t).$$

Here the environment dynamics $P$ and initial-state distribution $\mu$ are typically fixed, while the policy $\pi_\theta$ is the only factor that changes with $\theta$. This is the source of the gradient signal—but also the source of the technical obstacle. The parameter dependence is buried inside a product of probabilities, not inside a simple explicit formula for $R(\tau)$.

A tempting workaround is numerical differentiation. For the $i$-th parameter component, one could approximate

$$\frac{\partial J(\theta)}{\partial \theta_i} \approx \frac{J(\theta+\epsilon e_i)-J(\theta-\epsilon e_i)}{2\epsilon}.$$

This finite-difference estimate is conceptually straightforward, but it is computationally disastrous at scale. Each component of $\theta$ requires two fresh evaluations of the full objective, and each evaluation requires many rollouts because $R(\tau)$ itself is stochastic. For a neural policy with millions of parameters, that means millions of full rollouts per update in the worst case. The method is therefore not just slow; it is fundamentally mismatched to large-scale RL.

The real lesson is that we need a way to convert a gradient of an expectation into an expectation of a gradient-like quantity that can be estimated from sampled trajectories. In other words, we want to keep the Monte Carlo friendliness of sampling while still obtaining a valid derivative signal. That is the conceptual bridge to the likelihood ratio trick, which will let us rewrite the gradient without differentiating through the environment or through the return directly.

A useful way to keep the three roles separate is:

• $R(\tau)$ tells us how good a sampled trajectory was.
• $p(\tau\mid\theta)$ tells us how likely that trajectory was under the current policy.
• $\nabla_\theta J(\theta)$ must come from how changing $\theta$ changes the sampling distribution, not from differentiating rewards.

The visual below condenses exactly this bottleneck. The central equation emphasizes that the gradient is acting on an expectation, while the highlighted trajectory density reminds us that the parameter dependence lives inside the measure. The finite-difference box captures the brute-force alternative and why it scales so badly, and the bottom banner previews the escape hatch: once we apply the likelihood ratio trick, the gradient becomes something we can estimate from ordinary rollouts instead of from prohibitively many reruns.

We now have the core difficulty in focus: the objective is an expectation over trajectories, but those trajectories themselves depend on the policy parameters $\theta$. That means the derivative is not just acting on a familiar integrand $R(\tau)$; it is acting on the distribution $p(\tau \mid \theta)$. In other words, the obstacle is measure-theoretic rather than algebraic. If we try to differentiate the return directly through the environment, we immediately run into the fact that the dynamics may be unknown, discontinuous, or simply not differentiable in any useful way.

The key move is to rewrite the gradient in a form that isolates the parameter dependence inside a logarithm. For any differentiable density or mass function $p(\tau \mid \theta)$, the log-derivative identity says

$$\nabla_\theta p(\tau \mid \theta) = p(\tau \mid \theta)\,\nabla_\theta \log p(\tau \mid \theta).$$

This is just the chain rule in disguise: $\nabla_\theta \log p = \nabla_\theta p / p$, so multiplying through by $p$ recovers the original gradient. The point of this identity is not aesthetic; it is operational. It turns an awkward derivative of a probability measure into something that looks like an ordinary expectation under the same distribution.

Applying it to the return objective,

$$J(\theta) = \int R(\tau)\,p(\tau \mid \theta)\,d\tau,$$

we obtain

$$\nabla_\theta J(\theta) = \int R(\tau)\,\nabla_\theta p(\tau \mid \theta)\,d\tau = \int R(\tau)\,p(\tau \mid \theta)\,\nabla_\theta \log p(\tau \mid \theta)\,d\tau.$$

So the gradient is

$$\boxed{ \nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[ R(\tau)\,\nabla_\theta \log p(\tau \mid \theta) \right] }.$$

This is the essential policy-gradient transformation: the difficult derivative of an expectation becomes an expectation of a score function, i.e. the gradient of the log probability. Once written this way, the gradient can be estimated by Monte Carlo rollouts from the current policy, without differentiating through the environment’s state transitions.

A few subtle points are worth keeping in mind. First, this identity does not make the estimator low-variance by itself. In fact, the raw term $R(\tau)\,\nabla_\theta \log p(\tau \mid \theta)$ is often noisy because the return can fluctuate wildly across trajectories. That is why later we will introduce baselines and advantage functions: they preserve the expectation while reducing variance. Second, the derivation assumes we can exchange gradient and integral under mild regularity conditions; in practice this is usually justified for the smooth parameterizations used in policy networks, but it is still an assumption hiding in the background.

Conceptually, the result matters because it gives us a way to optimize policies in environments where the dynamics are unknown, stochastic, or non-differentiable. The only thing we need to differentiate is the policy’s own log-probability, which is exactly what neural networks can provide. That is why the score-function form is the foundation of REINFORCE, of baseline methods, and eventually of actor-critic algorithms. All of those methods are variations on this same expectation identity.

The visual below compresses that logic into a compact typographic progression. The top reminder isolates the original problem: the gradient is acting on a $\theta$-dependent measure. The middle identity box captures the whole trick in one line, with the short chain-rule proof underneath to show that nothing mysterious is happening. The bottom derivation then walks from the original integral to the Monte Carlo-friendly expectation, highlighting the exact point where the likelihood ratio form turns a hard calculus problem into a sampling problem.

Read as a whole, the figure is less a derivation than a map of the transformation: from a derivative that seems to require environmental gradients, to a score-function expectation that can be estimated from sampled trajectories. That transition is the conceptual hinge for everything that follows.

Having established the likelihood-ratio identity, the next question is what actually sits inside the trajectory score $\nabla_\theta \log p(\tau\mid\theta)$. This is the point where policy gradients become especially elegant: the trajectory probability factors into pieces generated by the environment and pieces generated by the policy, and only one of those depends on $\theta$.

For a finite-horizon trajectory $\tau=(s_0,a_0,s_1,a_1,\dots,s_T)$, the joint density under policy $\pi_\theta$ is

$$p(\tau\mid\theta) = \mu(s_0)\prod_{t=0}^{T-1} P(s_{t+1}\mid s_t,a_t)\,\pi_\theta(a_t\mid s_t).$$

Taking logs turns the product into a sum:

$$\log p(\tau\mid\theta) = \log \mu(s_0) + \sum_{t=0}^{T-1}\Bigl[\log P(s_{t+1}\mid s_t,a_t)+\log \pi_\theta(a_t\mid s_t)\Bigr].$$

This decomposition is not just algebraic bookkeeping. It encodes a crucial modeling assumption: the environment evolves according to dynamics $P(s' \mid s,a)$ that do not change when we adjust the policy parameters $\theta$. In other words, the policy chooses actions, the environment reacts, but the environment itself is not being differentiated through. That is exactly why the policy gradient method is called model-free: we do not need a differentiable model of the transition kernel to compute the gradient.

Now apply $\nabla_\theta$. The initial-state term $\log \mu(s_0)$ vanishes immediately, and so do all transition terms $\log P(s_{t+1}\mid s_t,a_t)$, because they are independent of $\theta$. What remains is just the sum of policy score functions:

$$\nabla_\theta \log p(\tau\mid\theta) = \sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta(a_t\mid s_t).$$

This is the heart of the derivation. Every time step contributes a local sensitivity term saying how the log-probability of the sampled action changes if we nudge the policy parameters. The environment still shapes the trajectory, but it contributes only through the realized states and rewards, not through any explicit gradient path.

Substituting this back into the likelihood-ratio form from the previous step yields the familiar REINFORCE estimator:

$$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[ R(\tau)\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta(A_t\mid S_t) \right].$$

A useful way to interpret this is as a correlation estimator: trajectories with larger return $R(\tau)$ reinforce the action choices that produced them. The estimator is unbiased, but it is also noisy because the same scalar return multiplies every action score in the trajectory. That variance problem is exactly what motivates baselines, reward-to-go, and actor-critic methods later on.

There is also a subtle but important boundary condition here. The cancellation of $P(s_{t+1}\mid s_t,a_t)$ relies on the standard policy-gradient setting where the environment dynamics are fixed with respect to $\theta$. If the policy parameters were to influence the dynamics directly, or if we were learning a differentiable simulator model jointly with the policy, then additional gradient paths could appear. In the classical RL setting, however, the result is clean: the only differentiable object inside the trajectory probability is the policy itself.

So the conceptual message is compact:

• Trajectory probability factorizes into initial state, environment transitions, and policy actions.
• Only the policy term depends on $\theta$.
• Therefore, the gradient is a sum of policy score functions.
• Consequently, the estimator is model-free.

The visual below condenses exactly that logic into three steps: expand the log trajectory probability, delete the $\theta$-independent terms, and arrive at the model-free policy gradient. It is less a new derivation than a proof-of-cancellation, and that cancellation is what makes policy gradient methods practical in unknown environments.

We can now make the policy gradient estimator a little more honest about when information becomes available. The previous form used the full trajectory return $R(\tau)$ with every score term $\nabla_\theta \log \pi_\theta(A_t\mid S_t)$, but that quietly mixes together rewards that happened before action $A_t$ with rewards that happen after it. Intuitively, that is wasteful: if a reward has already occurred, then the action at time $t$ could not have caused it.

This is where causality enters. In an MDP, action $A_t$ is selected after observing $S_t$, and only then does the environment transition forward. So when we ask how the parameters $\theta$ should change to increase expected return, the only rewards that should matter for the decision at time $t$ are the ones that lie in its causal future. Past rewards are fixed by the time the action is taken, so they cannot provide a useful learning signal for that action.

Formally, the key observation is a zero-mean identity. For any reward $R_k$ with $k \le t$, the score function term has no positive correlation with that past reward: $$\mathbb{E}_{\pi_\theta}\!\left[\nabla_\theta \log \pi_\theta(A_t \mid S_t)\cdot R_k\right] = 0 \quad \text{for all } k \le t.$$ This is not saying the gradient term is literally zero everywhere; it says that, in expectation, the contribution from rewards that happened before the action cancels out. That cancellation is exactly what lets us simplify the estimator without changing its mean.

So the full-trajectory return can be replaced by the future return from time $t$: $$G_t = \sum_{k=0}^{T-1-t} \gamma^k R_{t+k+1}.$$ With that substitution, the policy gradient becomes $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[\sum_{t=0}^{T-1} \nabla_\theta \log \pi_\theta(A_t \mid S_t)\cdot G_t\right].$$ This is the causal form of REINFORCE: each action is paired only with the rewards that it could plausibly influence.

The practical benefit is important. We have preserved unbiasedness because we only removed terms whose expectation is zero. But we have also improved the estimator’s variance. Every time we multiply a score term by the full return, we inject noise from rewards that the action could never affect. Removing those irrelevant terms usually makes the gradient signal sharper and learning more stable.

There is a subtle point worth emphasizing: this is not merely a bookkeeping trick. The same return $R(\tau)$ appears in every time step’s gradient estimate, but the relevant portion of that return depends on the time index. The estimator is therefore less like “score times total outcome” and more like “score times the consequences that lie ahead from this decision.” That causal interpretation is what makes the later actor-critic and advantage formulations feel natural rather than ad hoc.

A few takeaways make the logic compact:

• Past rewards are irrelevant to the current action’s credit assignment.
• Zero-mean terms can be dropped without changing the expected gradient.
• Future return $G_t$ is the correct signal for time $t$.
• Variance drops because we stop multiplying by noise from unrelated rewards.

The visual summary below organizes exactly this flow: the older full-return expression is shown as the starting point, the causal identity in the middle highlights why the past contributes nothing in expectation, and the final expression keeps only $G_t$. The small timeline on the side is especially useful because it turns the abstract proof into a picture: the red region marks rewards that are already in the past and therefore dropped, while the green region marks the future rewards that remain attached to the action.

Read that diagram as a compact proof sketch. The arrows are doing the same logical work as the equations: first identify the overcounted past, then invoke causality and zero expectation, and finally arrive at the lower-variance estimator that will power the rest of the policy gradient methods we build next.

Having established why only future rewards matter, we can now strip the problem down to its cleanest possible form and see the policy gradient mechanism with almost no distractions. The one-step MDP is the smallest setting where the entire idea is already present: there is a fixed start state $s_0$, we sample a single action $A \sim \pi_\theta(\cdot \mid s_0)$, and then immediately receive reward $r(s_0, A)$. There is no trajectory credit assignment yet, no return-to-go, and no bootstrapping—just a stochastic choice followed by a scalar payoff.

In this case, the objective is simply the expected reward under the policy: $$J(\theta) = \sum_{a \in \mathcal{A}} \pi_\theta(a \mid s_0)\, r(s_0, a).$$ This is the most direct expression of what policy optimization means. We are not trying to predict rewards explicitly; instead, we are adjusting the parameters so that the policy places more mass on actions that empirically lead to higher reward. The dependence on $\theta$ is entirely through the action probabilities.

The key step is to differentiate $J(\theta)$ with respect to $\theta$. Since $r(s_0,a)$ does not depend on the policy parameters, it passes through the gradient unchanged: $$\nabla_\theta J(\theta) = \sum_{a \in \mathcal{A}} \nabla_\theta \pi_\theta(a \mid s_0)\, r(s_0, a).$$ At this point the expression is correct but not yet useful, because it involves $\nabla_\theta \pi_\theta$ directly. The standard trick is to rewrite that derivative in terms of the log-probability: $$\nabla_\theta \pi_\theta(a \mid s_0) = \pi_\theta(a \mid s_0)\, \nabla_\theta \log \pi_\theta(a \mid s_0).$$ This is just the identity $\nabla p = p\,\nabla \log p$, sometimes called the likelihood ratio trick or log-derivative trick. Its power is that it converts a derivative of a probability into a probability-weighted derivative of a log-probability, which is exactly what we can estimate from samples.

Substituting that identity gives $$\nabla_\theta J(\theta) = \sum_{a \in \mathcal{A}} \pi_\theta(a \mid s_0)\, \nabla_\theta \log \pi_\theta(a \mid s_0)\, r(s_0, a).$$ Now the gradient is written as an expectation under the current policy: $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[\nabla_\theta \log \pi_\theta(A \mid s_0)\, r(s_0, A)\right].$$ This is the score function estimator in its most elementary form. The “score” is the gradient of the log policy, and the reward acts as the weight that says how strongly that sampled action should influence the update. High-reward actions push the policy up; low-reward actions push it down.

The intuition is worth stating carefully. Because $\nabla_\theta \log \pi_\theta(a \mid s_0)$ points in the direction that increases the log-probability of action $a$, multiplying by $r(s_0,a)$ means:

• actions with positive reward get reinforced,
• actions with negative reward get suppressed,
• and actions with larger magnitude reward produce stronger updates.

So the gradient ascent update $$\theta \leftarrow \theta + \alpha \nabla_\theta J(\theta)$$ literally “moves probability mass” toward the better actions. This is the first place where the probabilistic interpretation of policy gradients becomes concrete: we are not editing a value table, we are nudging a distribution.

There is also a subtle but important assumption hiding here: the reward term must be independent of $\theta$ for this derivation to hold in this simple form. In the one-step MDP, that is naturally true because the reward is treated as a function of the chosen action and state, not of the parameters directly. In more general settings, the same idea survives, but the algebra becomes richer because the action influences future states and rewards through the trajectory distribution.

The visual below compresses this chain into a single, readable flow: from the one-step setup, to the expected-reward objective, to the log-derivative rewrite, and finally to the boxed score function estimator. Read it as a compact proof sketch rather than a standalone formula sheet—the point is not just that the identities are true, but that each line prepares the next one by turning an awkward derivative of probabilities into a sample-friendly expectation.

We can now make the score-function update feel geometric rather than merely algebraic. The key question is not just what gradient ascent computes, but how a change in parameters moves the policy itself. For a discrete action space, the policy $\pi_\theta(\cdot \mid s)$ is a probability distribution over actions, so a gradient step should be understood as redistributing mass across actions, not simply nudging numbers in parameter space.

In the one-step setting, the objective is the expected reward $$J(\theta) = \mathbb{E}_{A \sim \pi_\theta(\cdot \mid s)}[R].$$ Applying the likelihood ratio trick gives $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[\nabla_\theta \log \pi_\theta(A \mid s)\, R\right].$$ This is already the essential geometric statement: each sampled action contributes a direction in parameter space, and that direction is weighted by how much reward followed that action. If the reward is positive, the update reinforces the probability of that action; if it is negative, the update suppresses it.

A useful way to think about this is to imagine each action carrying its own “vote” in the parameter update. The score function $\nabla_\theta \log \pi_\theta(a \mid s)$ tells us how to increase the probability of $a$ locally, and the reward $r(s,a)$ determines how loud that vote is. So the update is a reward-weighted sum of score directions, not an undifferentiated push toward every action equally.

This also explains why the update behaves sensibly in the presence of multiple actions with different payoffs. Suppose we start from a uniform policy over three actions, $\pi_\theta(a_i \mid s)\approx 0.33$, with rewards $r(s,a_1)=+2$, $r(s,a_2)=-1$, and $r(s,a_3)=+3$. Then a single gradient step should:

• increase $a_3$ the most, because it has the largest positive reward;
• increase $a_1$ somewhat, because it is good but less good than $a_3$;
• decrease $a_2$, because its reward is negative.

The important subtlety is that the policy update is relative. Because probabilities must still sum to one, increasing mass on high-reward actions necessarily takes mass away from lower-reward ones. This is why policy gradients are often described as pushing probability mass uphill on the reward landscape: the geometry lives on the simplex of categorical distributions, where improvement in one region implies contraction elsewhere.

There is also a deeper connection to reinforcement as a statistical learning principle. The update behaves like a policy-level analogue of Hebbian learning: actions that are “co-activated” with positive reward are strengthened, while actions associated with poor outcomes are weakened. Unlike supervised learning, however, the signal is not a target label; it is a scalar return that only tells us whether the sampled behavior was valuable.

Of course, this clean picture hides the fact that the raw Monte Carlo gradient can be noisy. A single sampled reward may be unrepresentative, and the score function can have large variance even when the direction is unbiased. That limitation is precisely why later variants introduce baselines and critics: they keep the same geometric rule—push mass toward better-than-average actions—while reducing the randomness in how hard each action is pushed.

The visual below compresses this intuition into a simple before-and-after story. Starting from equal probability bars, the update arrow points from the uniform policy to a new distribution where the tallest bar belongs to the highest-reward action, the intermediate reward gets a moderate increase, and the negative-reward action shrinks. The curved arrows are doing more than decorating the diagram: they encode the fact that the gradient acts like a mass transport mechanism, lifting probability where reward is high and draining it where reward is low.

Seen this way, the diagram is not just a summary of a particular example. It is a compact geometric rendering of $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[\nabla_\theta \log \pi_\theta(A \mid s)\cdot R\right],$$ and of the update $$\theta \leftarrow \theta + \alpha \sum_{a \in \mathcal{A}} \pi_\theta(a \mid s)\, r(s,a)\,\nabla_\theta \log \pi_\theta(a \mid s).$$ Together, they say the same thing: policy gradients do not directly optimize actions; they reshape probability mass so that better actions become more likely.

Building on the geometric view from before, we can now state the central identity that makes policy gradients practical: the gradient of performance can be written as an on-policy expectation of a local score term weighted by an action-value signal. This is the point where the optimization problem stops looking like “differentiate through the whole environment” and starts looking like “increase the probability of actions that turn out well.”

Formally, for a differentiable stochastic policy $\pi_\theta(a \mid s)$, define the return objective as the expected value under the policy’s own visitation distribution: $$J(\theta) = \sum_{s \in \mathcal{S}} d^{\pi_\theta}(s)\, V^{\pi_\theta}(s).$$ The notation matters here. The objective is not just a sum of per-state values; it is weighted by $d^{\pi_\theta}(s)$, the distribution over states the current policy actually visits. That dependence on $\theta$ is exactly what makes policy gradients seem tricky at first glance. If we differentiate $J(\theta)$ naively, it looks like we should have to backpropagate through both the value function and the state distribution. The theorem says something surprisingly cleaner happens.

The policy gradient theorem states that $$\nabla_\theta J(\theta) = \sum_{s \in \mathcal{S}} d^{\pi_\theta}(s) \sum_{a \in \mathcal{A}} Q^{\pi_\theta}(s,a)\,\nabla_\theta \pi_\theta(a \mid s).$$ Equivalently, using the score-function identity $\nabla_\theta \pi_\theta(a\mid s)=\pi_\theta(a\mid s)\nabla_\theta \log \pi_\theta(a\mid s)$, we obtain the compact expectation form $$\boxed{ \nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[ Q^{\pi_\theta}(S_t, A_t)\,\nabla_\theta \log \pi_\theta(A_t \mid S_t) \right] }.$$ This is the form that powers REINFORCE and essentially every modern policy-gradient method. It says: sample from the current policy, evaluate how good the sampled action was in hindsight, and nudge the policy parameters in the direction that makes that action a little more likely.

The subtle, non-trivial part is what does not appear. Although $d^{\pi_\theta}(s)$ depends on $\theta$, there is no explicit $\nabla_\theta d^{\pi_\theta}(s)$ term in the theorem. That omission is not a shortcut or approximation; it is an exact cancellation that follows from the dynamics of Markov decision processes. Intuitively, the effect of changing the policy on future state visitation is already accounted for indirectly through $Q^{\pi_\theta}(s,a)$, because the action-value function captures the downstream consequences of choosing $a$ in $s$. In other words, the theorem separates where the policy acts locally from how the environment propagates those choices forward.

This is why the theorem is so useful computationally. If the gradient had to explicitly differentiate through the visitation distribution, we would need a model of the environment dynamics or a full unrolled computation graph of the trajectory distribution. Instead, the theorem gives a model-free policy gradient that only requires sampled trajectories and estimates of $Q^{\pi_\theta}$ or returns. It applies both to episodic tasks, where $d^{\pi_\theta}$ can be read as a discounted occupancy measure, and to continuing tasks, where it is the stationary state distribution.

A few implications are worth keeping in view:

• Local score, global consequence: $\nabla_\theta \log \pi_\theta(a\mid s)$ says how to change the policy at the current state.
• Return weighting: $Q^{\pi_\theta}(s,a)$ tells us whether that action should be reinforced or discouraged.
• No dynamics derivative: the environment’s transition structure matters, but not through explicit gradient terms.
• On-policy expectation: the theorem is exact for samples coming from the same policy being optimized.

This theorem is the conceptual bridge between the intuitive “push probability mass toward good actions” picture and the concrete algorithms that follow. REINFORCE will replace $Q^{\pi_\theta}$ with sampled returns $G_t$; baselines will subtract variance-reducing terms without changing the expectation; actor-critic methods will learn a low-variance approximation to $Q^{\pi_\theta}$. All of those variations are easier to understand once this identity is in place, because they are all trying to estimate the same gradient in different ways.

The visual below is best read as a compact summary of that logic. The top portion collects the objective and the theorem statement in symbolic form, while the highlighted expectation form emphasizes the result we actually use in algorithms. The small notes at the bottom are there for two reasons: first, to underline that $\nabla_\theta d^{\pi_\theta}$ is absent despite seeming unavoidable; second, to remind us that the theorem covers both episodic and continuing settings. Taken together, the diagram condenses the whole message of the theorem: policy gradients are exact, on-policy, and local in form even though their effect is long-horizon and global in consequence.

We now have the statement of the policy gradient theorem; the remaining question is why it is true and, just as importantly, why the proof is structured the way it is. The key difficulty is that the objective $J(\theta)=V^{\pi_\theta}(s_0)$ depends on $\theta$ in two intertwined ways: directly through the policy $\pi_\theta$, and indirectly through the state distribution induced by repeatedly following that policy. A naive differentiation would appear to require tracking how the entire trajectory distribution changes with $\theta$, which looks hopeless.

The proof avoids that trap by working with the Bellman recursion for the value function. Starting from $$V^{\pi_\theta}(s)=\sum_a \pi_\theta(a\mid s)\,Q^{\pi_\theta}(s,a),$$ we apply the product rule: $$\nabla_\theta V^{\pi_\theta}(s) = \sum_a \Bigl[\nabla_\theta \pi_\theta(a\mid s)\,Q^{\pi_\theta}(s,a) +\pi_\theta(a\mid s)\,\nabla_\theta Q^{\pi_\theta}(s,a)\Bigr].$$ This split is the heart of the argument. The first term already has the shape we want: a policy gradient weighted by action value. The second term is the annoying recursive remainder, but it is also where the Bellman equation earns its keep.

To expand $\nabla_\theta Q^{\pi_\theta}(s,a)$, write the action-value function in one-step form: $$Q^{\pi_\theta}(s,a)=\sum_{s'}P(s'\mid s,a)\bigl[r+\gamma V^{\pi_\theta}(s')\bigr].$$ The transition kernel $P$ and reward are environment properties, so they do not depend on $\theta$. Only the downstream value $V^{\pi_\theta}(s')$ does. Hence $$\nabla_\theta Q^{\pi_\theta}(s,a) = \sum_{s'}P(s'\mid s,a)\,\gamma\,\nabla_\theta V^{\pi_\theta}(s').$$ This is the recursive step: the gradient of the value at one state is expressed through the gradient of the value at successor states, discounted by $\gamma$. If you keep substituting this relation forward, the gradient propagates along all possible future trajectories, with each path accumulating a factor of $\gamma^t$ and its corresponding probability under $\pi_\theta$.

That recursive unrolling produces a discounted visitation-weighted sum over states: $$\nabla_\theta V^{\pi_\theta}(s_0) = \sum_{s\in\mathcal S}\sum_{t=0}^\infty \gamma^t\,\Pr(s_0\to s,\,t,\,\pi_\theta) \sum_a \nabla_\theta \pi_\theta(a\mid s)\,Q^{\pi_\theta}(s,a).$$ It is useful to name the discounted occupancy measure $$\eta(s)=\sum_{t=0}^{\infty}\gamma^t\,\Pr(s_0\to s,\,t,\,\pi_\theta), \qquad d^{\pi_\theta}(s)=\frac{\eta(s)}{\sum_{s'}\eta(s')}.$$ This normalization is not just a cosmetic trick: it packages the infinite unrolling into a distribution over states visited under the policy. The important subtlety is that the derivative of this distribution never has to be computed. The proof does not differentiate the state occupancy measure directly; instead, it accumulates the contributions that flow through it, which is why the theorem is so practical.

At this point, the final simplification is the log-derivative trick: $$\nabla_\theta \pi_\theta(a\mid s) = \pi_\theta(a\mid s)\,\nabla_\theta \log \pi_\theta(a\mid s).$$ Substituting this converts the gradient into the familiar expectation form, $$\nabla_\theta J(\theta)\propto \mathbb E_{\pi_\theta}\!\left[ Q^{\pi_\theta}(S_t,A_t)\,\nabla_\theta \log \pi_\theta(A_t\mid S_t) \right].$$ The proportionality hides the normalization constant from $d^{\pi_\theta}$, but that constant is irrelevant for optimization because it does not change the direction of ascent. What matters is that the gradient can be estimated from sampled trajectories using only terms evaluated along the visited states and actions.

There are two conceptual payoffs here:

• No model of the policy-induced state distribution is needed.
• The gradient is expressed as an expectation, which is exactly the form we can estimate from samples.

That second point is what makes REINFORCE possible, and it also explains why the estimator is so noisy: the theorem gives us a correct unbiased direction, but it does not reduce variance by itself. Baselines, advantage functions, and actor-critic methods will all be refinements of this same identity, designed to replace the raw $Q$-signal with lower-variance surrogates while preserving unbiasedness.

The visual below condenses this proof into a compact chain: product rule $\rightarrow$ Bellman recursion $\rightarrow$ discounted unrolling $\rightarrow$ occupancy measure $\rightarrow$ log-derivative form. Read left to right, it emphasizes the one genuinely surprising fact in the proof: the gradient never needs a separate $\nabla_\theta d^{\pi_\theta}$ term. That absence is not a loophole; it is the theorem.

We now have the policy gradient theorem in its cleanest form, but the theorem by itself is still a little too abstract to implement directly. It tells us that the gradient can be written as an expectation involving the action-value function: $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[ Q^{\pi_\theta}(S_t, A_t)\,\nabla_\theta \log \pi_\theta(A_t \mid S_t) \right].$$ This is elegant, but it hides a practical problem: $Q^{\pi_\theta}(S_t, A_t)$ is itself unknown. To use the theorem in code, we need some quantity we can actually compute from experience.

The simplest candidate is the Monte Carlo return $G_t$, the sampled sum of future rewards from time $t$ onward. The key identity is that, by definition of the action-value function, $$Q^{\pi_\theta}(s, a) = \mathbb{E}_{\pi_\theta}\!\left[ G_t \mid S_t = s,\, A_t = a \right].$$ This says that $Q^{\pi_\theta}(s,a)$ is not some unrelated target; it is exactly the conditional expectation of the return after taking action $a$ in state $s$ and then following the policy. So when we observe one rollout and compute a realized return $G_t$, we are drawing one sample from the distribution whose mean is $Q^{\pi_\theta}(S_t, A_t)$.

That observation justifies the Monte Carlo replacement through iterated expectation. If we multiply by the score function $\nabla_\theta \log \pi_\theta(A_t \mid S_t)$ and then average over trajectories, the sampled return and the true action-value produce the same expected gradient: $$\mathbb{E}_{\pi_\theta}\!\left[ G_t\,\nabla_\theta \log \pi_\theta(A_t \mid S_t) \right] = \mathbb{E}_{\pi_\theta}\!\left[ Q^{\pi_\theta}(S_t, A_t)\,\nabla_\theta \log \pi_\theta(A_t \mid S_t) \right] = \nabla_\theta J(\theta).$$ The subtle point is that the replacement is not “approximately true” in expectation; it is exactly unbiased as long as $G_t$ is sampled from the correct policy-induced trajectory distribution. This is why Monte Carlo policy gradients are so attractive: they require no model of the environment and no bootstrapping assumptions.

There is, however, an important cost. $G_t$ is a random variable with potentially large spread, especially in long-horizon tasks where each reward is a noisy proxy for eventual success. The estimator is unbiased, but unbiasedness alone does not guarantee usefulness. In practice, a very noisy gradient can make learning unstable, slow, or even appear to fail entirely because updates point in inconsistent directions from episode to episode.

This is the first place where the broader design space of policy gradients becomes visible. The theorem does not demand that we use the exact function $Q^{\pi_\theta}$; it only demands an unbiased estimator of it. So any $\hat{Q}_t$ satisfying $$\mathbb{E}_{\pi_\theta}[\hat{Q}_t \mid S_t, A_t] = Q^{\pi_\theta}(S_t, A_t)$$ can be plugged into the same gradient formula without changing the expectation of the update. That freedom is the conceptual bridge to baselines and critics: once you understand that the theorem only cares about the mean of the estimator, you can start trading computation structure for lower variance.

From this viewpoint, Monte Carlo returns are the most direct substitute:

• Pros: unbiased, model-free, easy to derive
• Cons: high variance, often requires full episodes

Later, a critic will try to estimate $Q$ or $V$ more smoothly than raw returns, while baselines will subtract a control variate that leaves the mean gradient unchanged. But those are refinements of the same basic logic: replace the unknown $Q$ with something whose conditional expectation matches it.

The visual below compresses that argument into three steps. The top blocks start with the theorem, then identify $G_t$ as a conditional expectation of $Q^{\pi_\theta}$, and finally show the substitution justified by iterated expectation. The lower comparison area highlights the real tradeoff: we gain a simple unbiased estimator, but we pay for it with variance. Read as a whole, the diagram is less a new result than a compact proof sketch that explains why the Monte Carlo form is valid and why it is only the beginning of the story.

Up to this point, the policy-gradient story has been about replacing an intractable derivative of environment dynamics with something we can estimate from sampled experience. REINFORCE is the most direct expression of that idea: it takes the Monte Carlo return and uses it as a learning signal for the policy’s score function. In other words, instead of asking, “What action is optimal in this state?” it asks, “Which sampled actions seemed to lead to better returns, and how should I increase their probability?”

The core estimator is

$$\nabla_\theta J(\theta) \approx \sum_{t=0}^{T-1} G_t \, \nabla_\theta \log \pi_\theta(A_t \mid S_t),$$

where the return from time $t$ is

$$G_t = \sum_{k=t}^{T-1} \gamma^{k-t} R_{k+1}.$$

This is the most literal Monte Carlo policy gradient: for each visited state-action pair, we compute the total discounted reward that followed it, and then push the parameters in the direction that makes that action more likely. The score function term $\nabla_\theta \log \pi_\theta(A_t \mid S_t)$ says how to change the policy locally; the scalar return $G_t$ says whether that sampled decision was good or bad.

A useful way to think about the update is as a form of credit assignment by hindsight. If an action eventually leads to a large return, then the update increases its probability; if it leads to a poor return, the same mechanism decreases its probability. The parameter step is

$$\theta \leftarrow \theta + \alpha \, G_t \, \nabla_\theta \log \pi_\theta(A_t \mid S_t),$$

which is just stochastic gradient ascent on expected return. The elegance here is that the update never needs a model of the environment and never needs an explicit $Q$-function approximation. It only needs sampled trajectories and the policy’s own log-probability gradient.

There is, however, an important subtlety: the update is unbiased only because $G_t$ is a Monte Carlo sample of the action-value under the current policy. That means REINFORCE is tied to the data distribution generated by $\pi_\theta$ itself. If the episode came from some different behavior policy, then the estimator would no longer be the plain on-policy form; one would need importance sampling corrections. For the basic algorithm, the price of simplicity is that we must collect full trajectories under the current policy before making updates.

That episodic requirement matters practically. Because $G_t$ depends on rewards that occur after time $t$, we cannot update immediately after a single step without either waiting for the episode to finish or introducing additional bootstrapping machinery. This makes REINFORCE conceptually clean but often sample-inefficient: the whole trajectory is used as a delayed training signal, and every action in the episode receives a return-based weight. In long-horizon tasks, those weights can be noisy, especially when many actions have only a weak influence on the final outcome.

The estimator can also be written as a sum of per-time-step gradients, which makes its mechanism easier to remember:

• direction: $\nabla_\theta \log \pi_\theta(A_t \mid S_t)$
• weight: $G_t$
• step size: $\alpha$

This decomposition is the bridge between the theory and the implementation. The gradient term tells us how to change the policy distribution; the return tells us which sampled decisions deserve reinforcement. Nothing in the update requires a critic, a value baseline, or temporal-difference bootstrapping yet.

The visual below is a compact summary of exactly that logic. The equation at the top captures the policy-gradient estimator in its Monte Carlo form, while the pseudocode box turns the math into an operational recipe: generate a full episode, compute $G_t$ for each time step, and apply the score-function update immediately afterward. The three short callouts reinforce the essential properties that make REINFORCE both attractive and problematic: it is on-policy, unbiased, and episodic.

That combination is what makes REINFORCE the canonical starting point for variance reduction. It is the simplest correct policy-gradient algorithm, and precisely because it is so direct, its weaknesses become easy to see. Those weaknesses are not a flaw in the derivation; they are the inevitable consequence of using raw Monte Carlo returns as the learning signal.

After establishing that REINFORCE gives us an unbiased policy gradient estimate, the next question is the one that matters in practice: why does it still feel so unstable? The answer is variance. Unbiasedness only says that, in expectation, the estimator points in the right direction; it says nothing about how wildly individual samples may deviate from that direction. In policy gradient methods, those deviations can be so large that learning becomes painfully slow, even though the underlying objective is theoretically correct.

The core estimator is

$$\nabla_\theta J(\theta) \approx \frac{1}{N}\sum_{n=1}^{N}\sum_{t=0}^{T-1} G_t^{(n)} \nabla_\theta \log \pi_\theta(A_t^{(n)} \mid S_t^{(n)}).$$

This looks deceptively simple: each action gets weighted by the return observed after it, and the average over many rollouts approximates the true gradient. But the quantity doing the heavy lifting here is $G_t$, the Monte Carlo return from time $t$,

$$G_t = \sum_{k=0}^{T-t-1} \gamma^k R_{t+k+1}.$$

That sum is where the trouble begins. It aggregates a long chain of stochastic rewards, and the longer the horizon, the more random terms accumulate. A useful mental model is that the variance of $G_t$ grows roughly with the remaining episode length:

$$\operatorname{Var}(G_t) \propto (T - t).$$

So early actions are evaluated using the noisiest return estimates, because they depend on the entire future of the trajectory. In environments like CartPole-v1, where an episode can last 200 steps, the first few updates are effectively tied to sums of hundreds of random reward contributions.

There are really two different sources of noise here. First, trajectory stochasticity: even if we start from the same policy, small differences in sampled actions cause trajectories to diverge, and those divergences compound over time. Second, temporal credit misassignment: the return $G_t$ includes rewards that happened after action $A_t$, even when that action had little or no causal influence on them. The gradient update therefore treats distant rewards as evidence about nearby decisions, which is statistically legal but often semantically misleading.

This distinction is important. The estimator remains unbiased because, in expectation, the log-derivative trick correctly attributes score to the policy's chosen actions. But high variance means a single episode can push the parameters in a very different direction from the next episode, even when both are sampled from the same policy. In effect, the learning signal is correct on average but too noisy to be useful without some form of variance reduction.

That is why REINFORCE often converges so slowly in practice. On CartPole-v1, vanilla Monte Carlo policy gradients may require more than 10,000 episodes to become reliable, while methods that reduce variance can reach the same performance in under 1,000. The difference is not that the objective changes; it is that the estimator becomes less distracted by irrelevant fluctuations. The most direct fix is to subtract a baseline $b(s)$, ideally one that depends on the state but not on the sampled action. Intuitively, this recenters the learning signal so that only relative advantage matters.

A few takeaways are worth keeping in mind:

• Unbiased does not mean practical.
• Longer horizons amplify return variance.
• Distant rewards often inject noise into local credit assignment.
• Baselines help by removing predictable variation, not by changing the target.

The visual below compresses these ideas into two complementary pictures. On the left, the trajectory tree makes the credit problem concrete: early actions must “inherit” a noisy return built from many future rewards, and the red distant nodes emphasize how much of that signal is only weakly connected to the action being updated. On the right, the learning curves turn the statistics into an empirical story: the same algorithm that is mathematically sound can still crawl, while a variance-reduced version climbs much faster toward the optimal return. Together, they motivate the next step: subtracting a baseline to preserve correctness while taming the noise.

We now have the key variance-reduction idea in hand: subtract something that helps numerically, but contributes nothing in expectation. In policy gradients, that “something” is a baseline $b(s)$ that depends only on the state. The subtle point is that this is not an ad hoc trick—it is an exact identity that follows from the way the policy gradient is constructed.

Start from the REINFORCE form of the gradient, where the policy is weighted by a return-like signal. If we replace $Q^{\pi_\theta}(S_t,A_t)$ with $Q^{\pi_\theta}(S_t,A_t)-b(S_t)$, the gradient remains unchanged as long as $b$ does not depend on the sampled action. The reason is that the expected contribution of the baseline vanishes: $$\mathbb{E}_{\pi_\theta}\!\left[\nabla_\theta \log \pi_\theta(A_t \mid S_t)\, b(S_t)\right] = 0.$$ This identity is the heart of the theorem. It says the baseline can reduce variance without shifting the mean of the estimator, so we are not trading bias for stability. We are only changing the spread of the Monte Carlo signal.

Why does the expectation collapse to zero? Condition on the state $S_t=s$. Then $b(s)$ is just a constant with respect to the action draw $A_t \sim \pi_\theta(\cdot \mid s)$, so $$\mathbb{E}_{A_t \sim \pi_\theta(\cdot \mid s)}\!\left[\nabla_\theta \log \pi_\theta(A_t \mid s)\, b(s)\right] = b(s)\,\nabla_\theta \sum_a \pi_\theta(a\mid s) = b(s)\,\nabla_\theta 1 = 0,$$ assuming a discrete action space; the continuous-action version uses the same normalization argument with an integral. This is the quiet but important structural fact behind baseline methods: the score function $\nabla_\theta \log \pi_\theta(a\mid s)$ always integrates to zero under the policy that generated the sample.

Once that identity is in place, the policy gradient theorem immediately admits the centered form $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[\bigl(Q^{\pi_\theta}(S_t, A_t)-b(S_t)\bigr)\,\nabla_\theta \log \pi_\theta(A_t \mid S_t)\right].$$ This is the same gradient, just rewritten with a better-conditioned learning signal. Intuitively, the policy no longer asks, “Was this return large in absolute terms?” It asks, “Was this action better or worse than what I would typically expect from this state?”

That interpretation becomes especially clean with the canonical choice $b(s)=V^{\pi_\theta}(s)$. Then the centered return becomes the advantage function $$A^{\pi_\theta}(s,a)=Q^{\pi_\theta}(s,a)-V^{\pi_\theta}(s),$$ and the gradient takes the compact form $$\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\!\left[A^{\pi_\theta}(S_t,A_t)\,\nabla_\theta \log \pi_\theta(A_t \mid S_t)\right].$$ This is conceptually important because it separates two roles that REINFORCE had been mixing together:

• $V^{\pi_\theta}(s)$ estimates what is normal in state $s$.
• $A^{\pi_\theta}(s,a)$ measures whether action $a$ is better than normal.

The variance reduction matters because the raw return can be dominated by environment noise, horizon length, and reward scale, all of which obscure the learning signal. A baseline removes the predictable state-dependent component, often making updates much smaller in variance and therefore much more stable. But the theorem also clarifies a common failure mode: if the baseline depends on the action, then the zero-expectation argument breaks, and bias can creep in unless the correction is handled carefully.

So the real lesson is not merely that baselines are “allowed,” but that they are mathematically free so long as they remain state-only. That freedom is what lets us build actor-critic methods: the critic learns a baseline, usually $V^{\pi_\theta}(s)$, while the actor receives the advantage-weighted policy gradient. The result is a more efficient estimator that preserves the exact objective.

The visual below compresses this argument into a compact theorem-style layout. The upper block states the zero-bias identity itself, while the lower equations show the two natural rewritings: first with an arbitrary baseline $b(s)$, and then with the canonical choice $b(s)=V^{\pi_\theta}(s)$. Read together, they make the logic almost mechanical: subtract a state-only function, keep the same expected gradient, and interpret the remainder as advantage.

We now have the key algebraic ingredient that makes baselines useful in policy gradients: they can reduce variance without changing the expected update. The essential point is that a baseline must depend only on the state $s$, not on the sampled action $a$. That restriction is what turns the baseline term into something that averages to zero under the policy.

To see why, fix a state $s$ and consider the random action $A \sim \pi_\theta(\cdot \mid s)$. The baseline contribution to the policy-gradient estimator is $$\mathbb{E}_{A \sim \pi_\theta(\cdot \mid s)}\!\left[\nabla_\theta \log \pi_\theta(A \mid s)\, b(s)\right].$$ At first glance this looks nontrivial, because $\nabla_\theta \log \pi_\theta(A \mid s)$ is a random vector that depends on the sampled action. But $b(s)$ is fixed once the state is fixed, so it can be pulled outside the expectation. The remaining expectation is just the expected score function under the policy.

Now the important identity enters: for discrete actions, $$\nabla_\theta \log \pi_\theta(a \mid s)=\frac{\nabla_\theta \pi_\theta(a \mid s)}{\pi_\theta(a \mid s)}.$$ Substituting this into the sum over actions causes the policy probability to cancel, leaving $$b(s)\sum_a \nabla_\theta \pi_\theta(a \mid s).$$ This is where normalization does the real work. Since $\sum_a \pi_\theta(a \mid s)=1$ for every $\theta$, differentiating both sides gives $$\sum_a \nabla_\theta \pi_\theta(a \mid s)=\nabla_\theta 1=\mathbf{0}.$$ So the whole baseline term vanishes in expectation.

That cancellation is the reason baselines are so attractive in REINFORCE-style methods: they can change the spread of the estimator without changing its mean. Put differently, the baseline acts as a control variate whose expectation is exactly zero under the current policy. The policy-gradient estimate stays unbiased, but its variance can drop dramatically if $b(s)$ tracks the typical return from that state.

There is also a subtle failure mode worth keeping in mind. The argument breaks the moment the baseline depends on the action. If $b$ were $b(s,a)$, then it would no longer factor out of the expectation, and the $\pi_\theta(a\mid s)$ cancellation would not go through. In that case the “baseline” would generally introduce bias rather than remove variance. This is why, in policy-gradient practice, baselines are usually implemented as state-value functions $V^\pi(s)$ or approximations to them.

This proof also explains why advantage functions are so natural. If we write $$A^\pi(s,a)=Q^\pi(s,a)-V^\pi(s),$$ then the subtraction of $V^\pi(s)$ is exactly the kind of zero-mean baseline we just proved harmless. The update can then use the advantage instead of the raw return, which tends to center the learning signal around what is better or worse than expected rather than what is merely large in absolute terms.

The visual below compresses that algebra into a compact chain of equalities: factor out the state-only baseline, rewrite the score function with the log-derivative identity, cancel the policy in numerator and denominator, and finally invoke normalization to reach zero. The boxed remark at the bottom highlights the only real constraint in the argument: action-independence. Once that is clear, the result becomes a reusable lemma for the next step, where we turn this identity into the practical REINFORCE with baseline estimator.

After the zero-bias baseline identity, the natural next step is to ask: if any state-dependent baseline is allowed, what baseline should we actually learn? The answer is to estimate the state value $V^\pi(s)$, because it is the most obvious quantity whose scale matches the return and therefore can remove a large fraction of the variance in the policy-gradient signal. In practice we do not know $V^\pi$, so we fit a parametric approximation $\hat{V}_w(s)$ and use it as a learned baseline.

This gives the familiar advantage-style update $$\delta = G_t - \hat{V}_w(S_t),$$ where $G_t$ is the Monte Carlo return from time $t$, $$G_t = \sum_{k=t}^{T-1} \gamma^{k-t} R_{k+1}.$$ The quantity $\delta$ is not just a heuristic error term: it is the return centered by the critic’s prediction. If $\hat{V}_w$ is accurate, then $\delta$ is small for actions whose outcomes were ordinary and large in magnitude for unexpectedly good or bad outcomes. That makes the policy update much less noisy than plain REINFORCE, while preserving the same expected gradient as long as the baseline depends only on the state.

The algorithm therefore has two coupled learning problems. The critic learns to predict returns by minimizing a squared-error objective, while the actor uses the resulting advantage estimate to update the policy. The critic step is $$w \leftarrow w + \alpha_w \, \delta \, \nabla_w \hat{V}_w(S_t),$$ which is just stochastic gradient ascent on $-\tfrac12 (G_t - \hat{V}_w(S_t))^2$ or equivalently gradient descent on the prediction error. The actor step is $$\theta \leftarrow \theta + \alpha_\theta \, \gamma^t \, \delta \, \nabla_\theta \log \pi_\theta(A_t \mid S_t).$$ Here the same $\delta$ drives the policy update, but now it is weighted by the score function $\nabla_\theta \log \pi_\theta(A_t \mid S_t)$, which tells us how to change the probability of the sampled action. Positive advantage increases the action’s log-probability; negative advantage decreases it.

A subtle but important detail is the factor $\gamma^t$. In episodic problems, the policy gradient theorem can be written as an expectation over discounted state visitation, and that discounting appears in the per-step contribution when we expand the gradient into an episode-wise sum. If you omit it in this formulation, you implicitly change the weighting of earlier versus later decisions. It is one of those bookkeeping factors that is easy to overlook but essential for matching the theoretical objective.

The resulting method is often described as REINFORCE with baseline, but it is more revealing to think of it as the first actor-critic algorithm in miniature:

• the actor samples actions from $\pi_\theta$,
• the critic estimates how good the current state is,
• the advantage tells the actor whether the sampled action beat expectations.

This is precisely why the method reduces variance without introducing bias: subtracting $\hat{V}_w(S_t)$ changes the center of the learning signal, not its expectation, provided the baseline is state-dependent and not action-dependent. In other words, the critic is allowed to reshape the noise, but not to change which direction is correct in expectation.

There is also a practical reason to keep the critic and actor learning rates separate. The critic should usually track the moving target $V^{\pi_\theta}$ quickly enough to stay informative, but not so aggressively that it becomes unstable. The actor, meanwhile, should move more cautiously because it is optimizing the actual policy objective. When the critic is undertrained, $\delta$ remains noisy; when it is overconfident but inaccurate, the actor can be pushed in poor directions. So the two-step update is simple, but the interaction is delicate.

The visual below compresses all of that into a compact algorithmic picture: the top reminds you that the method is still REINFORCE at heart, the center shows how the return is turned into an advantage via the learned value baseline, and the highlighted lines separate the critic update from the actor update. That split is the key conceptual move. Once you can read the pseudo-code as “estimate value, form advantage, update critic, update policy,” the whole method becomes easy to place inside the broader bias-variance story.

It also prepares the ground for what comes next. REINFORCE with baseline still uses full Monte Carlo returns, so it keeps the low-bias, high-variance character of sampling complete episodes. The baseline makes the signal cleaner, but it does not yet bootstrap from incomplete future predictions. That is exactly the transition point to the next idea: how to trade Monte Carlo targets for bootstrapped ones in order to push variance down further, at the cost of introducing controlled bias.

After introducing a baseline, the natural next question is: why stop at subtracting variance from a Monte Carlo return if we can estimate the return itself more aggressively? The answer is the core idea behind actor-critic methods. Instead of waiting until the end of the trajectory to compute the full return $G_t$, we let a learned value function supply a bootstrapped estimate of the future. That move replaces “sum everything you will ever see” with “use what you have now, plus a guess about what comes next.”

For REINFORCE, the return is

$$G_t = \sum_{k=0}^{T-t} \gamma^k R_{t+k+1},$$

which is conceptually clean: if you sample complete trajectories from the current policy, $G_t$ is an unbiased target for $Q^{\pi_\theta}(S_t, A_t)$. But that unbiasedness comes at a price. Every additional future reward term injects randomness, so as the horizon grows, the target becomes increasingly noisy. In long episodes, the policy gradient is then driven by a signal that is correct on average but volatile from sample to sample.

Bootstrapping changes the estimator from “roll out all the way to the end” to “stop after one step and consult a critic.” A common one-step target is

$$\hat{Q}(S_t, A_t) = R_{t+1} + \gamma\,\hat{V}_w(S_{t+1}).$$

This is a much smaller random object: only the immediate reward is sampled directly, while the rest of the future is summarized by $\hat{V}_w$. The variance drops sharply because we no longer accumulate a long chain of stochastic rewards. The tradeoff is that $\hat{V}_w$ is only an approximation, so $\hat{Q}$ is generally biased relative to the true action-value function. In other words, we are no longer guaranteed to point exactly toward the true gradient direction at every step.

The same idea can be expressed even more locally through the TD error

$$\delta_t = R_{t+1} + \gamma\,\hat{V}_w(S_{t+1}) - \hat{V}_w(S_t).$$

This quantity is best understood as an advantage-like correction: if the observed outcome is better than the critic predicted, $\delta_t > 0$ and the actor increases the probability of the chosen action; if worse, the probability is decreased. If the critic were perfect, meaning $\hat{V}_w = V^{\pi_\theta}$, then $\delta_t$ would be an unbiased sample of the advantage $A^{\pi_\theta}(S_t, A_t)$. In practice, the critic is imperfect, so the update is biased—but often only mildly so, and that small bias is a worthwhile payment for a much cleaner learning signal.

This is why the actor update becomes

$$\theta \leftarrow \theta + \alpha\,\delta_t\,\nabla_\theta \log \pi_\theta(A_t \mid S_t).$$

Notice the structural beauty here: the policy gradient still has the same likelihood-ratio form, but the return estimate has been replaced by a critic-generated teaching signal. The actor does not need a full Monte Carlo return anymore; it only needs a scalar that says whether the recent action was better or worse than expected. Meanwhile, the critic is trained to reduce its own prediction error, so the two components co-evolve.

The essential lesson is the bias-variance tradeoff:

• Monte Carlo targets are low-bias but high-variance.
• Bootstrapped targets are higher-bias but much lower-variance.
• In many control problems, a small amount of bias is a very good deal if it yields faster, more stable learning.

The visual below is useful precisely because it compresses that tradeoff into a single comparison. The left side emphasizes how REINFORCE spreads credit across an entire reward sequence, which is statistically faithful but noisy. The right side shows the actor-critic compromise: one immediate reward, one bootstrapped estimate, and a TD error that serves as the actor’s training signal. Read together, the two panels make the central point tangible—actor-critic methods are not abandoning policy gradients, but approximating their targets in a way that dramatically lowers variance while accepting controlled bias.

Building on the bias–variance tradeoff, the actor–critic architecture makes a very practical compromise: instead of waiting for a full return $G_t$ to score an action, we let a learned critic provide a fast, local evaluation signal to a learned actor that controls behavior. This keeps the policy-gradient machinery intact, but replaces the noisy Monte Carlo target with something that can be formed at every step.

The key idea is that we now maintain two separate parameter vectors with different jobs. The actor is the policy $\pi_\theta(a \mid s)$, and its only responsibility is to choose actions that increase the objective $J(\theta)$. The critic is a value approximator $\hat{V}_w(s) \approx V^{\pi_\theta}(s)$, whose job is to estimate how good the current state is under the current policy. Because these components learn different quantities, they do not share parameters: $\theta$ is updated to improve control, while $w$ is updated to improve prediction.

What makes this architecture interesting is that the critic does not merely predict state values for their own sake; it also produces an advantage-like signal for the actor. The one-step temporal-difference error is $$\delta_t = R_{t+1} + \gamma \hat{V}_w(S_{t+1}) - \hat{V}_w(S_t).$$ If $\delta_t > 0$, the outcome was better than the critic expected, so the taken action should be reinforced. If $\delta_t < 0$, the outcome was worse than expected, so the actor should reduce the probability of repeating that action in that state. In that sense, $\delta_t$ plays the same conceptual role as an advantage estimate.

The critic itself is usually trained by a semi-gradient TD(0) update: $$w \leftarrow w + \alpha_w\,\delta_t\,\nabla_w \hat{V}_w(S_t).$$ This is a prediction problem, not a control problem. The critic tries to make its current estimate agree with a bootstrapped target $R_{t+1} + \gamma \hat{V}_w(S_{t+1})$, which means it can learn from a single transition rather than waiting for episode termination. That is the source of the lower variance: we are no longer averaging over long return trajectories.