In the previous essay we spent a long time with the Fokker–Planck equation, the law obeyed by the probability density of anything that drifts and jitters, and we ended on a theorem: diffusion is the steepest descent of a free energy, in the geometry of optimal transport. That essay was about prediction. Given where the crowd is now, where will it be?
This essay is about the other question, the one engineers and animals actually care about: given where I want to end up, what should I do? That is a control problem, and control problems are governed by an equation we have not met, the Hamilton–Jacobi–Bellman equation, which is nonlinear, backward in time, and famously hard. The line above the title says what this essay will prove about it: for a large and natural class of problems, one logarithm turns the hard nonlinear equation of wanting into the easy linear equation of expecting. Solving for the optimal action becomes computing a conditional expectation. Planning becomes inference.
That slogan gets said often and vaguely. Our job is to earn it: to state exactly which problems it holds for, to derive the transform in full, to watch it run, and to name the price. Along the way the main characters of the previous essay will keep walking on stage uninvited: the score \(\nabla\log\) of something will turn out to be the optimal action, the value function will turn out to be a free energy \(-\lambda\log Z\), and the optimal policy will turn out to be a Boltzmann distribution. As before, underlined terms expand into derivations and definitions, and the figures are the argument.
We need one recap and one new object. The recap: our system is the Langevin diffusion \(\mathrm dX_t = b(X_t)\,\mathrm dt + \sqrt{2T}\,\mathrm dW_t\), with drift \(b=-\nabla V\), and everything about it is encoded in its
For a Markov process, \((\mathcal L\varphi)(x) = \lim_{s\downarrow 0}\frac{\mathbb E_x[\varphi(X_s)]-\varphi(x)}{s}\): the instantaneous rate of change of the expectation of an observable \(\varphi\), started at \(x\). For our diffusion, Itô's lemma gave \(\mathcal L = b\cdot\nabla + T\Delta\). The generator determines the process completely; every question below is a question about this one operator.
In the previous essay we used its adjoint: probability densities evolve by \(\partial_t\rho = \mathcal L^{*}\rho\), and writing \(\mathcal L^{*}\) out is the Fokker–Planck equation. Kolmogorov called that the forward equation, because it evolves the present forward. The name implies a sibling.
$$\mathcal L \;=\; b\cdot\nabla \;+\; T\,\Delta,$$
the operator whose adjoint drives the Fokker–Planck equation. We called Fokker–Planck by its probabilist's name once, Kolmogorov's forward equation, and moved on. The name was a loaded gun. Forward of what? There is a backward equation, it is the one this essay lives in, and it answers a different kind of question.
Fix a future time \(T_{\!f}\) and a payoff \(\Phi\) to be evaluated then, and define
$$u(x,t) \;=\; \mathbb E\big[\,\Phi(X_{T_{\!f}}) \,\big|\, X_t = x\,\big],$$
the expected payoff given that we stand at \(x\) at time \(t\). This is a function of the present, about the future. A short shows it satisfies the Kolmogorov backward equation,
Condition on the first instant. Standing at \(x\) at time \(t\), the process spends \(\mathrm ds\) diffusing to \(X_{t+\mathrm ds}\), after which the expected payoff is \(u(X_{t+\mathrm ds}, t+\mathrm ds)\). The tower property of conditional expectation says these must agree:
$$u(x,t) \;=\; \mathbb E_x\big[u(X_{t+\mathrm ds},\, t+\mathrm ds)\big].$$
Expand the right side to first order: the time argument contributes \(\partial_t u\,\mathrm ds\), and the space argument contributes \(\mathbb E_x[u(X_{t+\mathrm ds},t)] - u(x,t) = (\mathcal Lu)\,\mathrm ds\), by the definition of the generator. The zeroth-order terms cancel, and what remains is \(\partial_t u + \mathcal Lu = 0\).
Note the logical difference from the forward derivation. There, we pushed a density through time and needed the adjoint. Here the generator acts directly, no integration by parts, no adjoint: the backward equation is the generator's native habitat, and Fokker–Planck is its mirror image.
$$\frac{\partial u}{\partial t} \;+\; \mathcal L u \;=\; 0, \qquad u(x, T_{\!f}) = \Phi(x),$$
with the data given at the end and the equation solved toward the present. The two Kolmogorov equations are exact duals: the forward one evolves what is (a density, flowing out of the past), the backward one evolves what to expect (a value, flowing out of the future), and the pairing between them, , is constant in time, because it equals the one number both equations are bookkeeping for: \(\mathbb E[\Phi(X_{T_{\!f}})]\). The first figure puts this conservation law on screen, and it is worth a minute of scrubbing to feel how strange and clean it is: two curves, moving in opposite temporal directions for opposite reasons, whose overlap never moves.
Differentiate the pairing and substitute both equations:
$$\frac{\mathrm d}{\mathrm dt}\!\int u\,\rho \;=\; \int (\partial_t u)\,\rho + \int u\,(\partial_t\rho) \;=\; -\!\int (\mathcal Lu)\,\rho + \int u\,(\mathcal L^{*}\rho) \;=\; 0,$$
the last step being the definition of the adjoint. Probabilistically the constant is obvious: \(\int u(x,t)\rho(x,t)\,\mathrm dx = \mathbb E\big[\mathbb E[\Phi(X_{T_f})|X_t]\big] = \mathbb E[\Phi(X_{T_f})]\), which does not depend on the intermediate time \(t\) we chose to condition at. The tower property, drawn as a conservation law.
In the figure, the discretization is chosen so this holds to machine precision: the backward solver is the exact matrix transpose of the forward solver from the previous essay. Duality is not approximated; it is inherited.
The blue density flows out of the past; the green expectation flows out of the future, from the payoff you draw. Scrub time and watch the readout: the overlap \(\int u\rho\) does not move, to thirteen decimal places, because both curves are accounts of the same number.
Before control enters, take stock of how much the backward equation already owns. Any question of the form "starting from here, what is the expected value of something later" is a backward equation: hitting probabilities, expected exit times, ruin probabilities, and the , which extends it to payoffs accumulated along the way and is, among other things, how options are priced (the Black–Scholes equation is a Kolmogorov backward equation in light disguise) and how quantum mechanics in imaginary time becomes a statement about Brownian paths. The forward equation of the previous essay told us what the world does. The backward equation is the natural home of every question about consequences, and consequences are what control is made of.
Add a running rate: let \(q(x)\ge 0\) be a cost accrued per unit time, and define
$$\psi(x,t) \;=\; \mathbb E\Big[\, e^{-\frac{1}{\lambda}\int_t^{T_f} q(X_s)\,\mathrm ds}\; e^{-\frac{1}{\lambda}\Phi(X_{T_f})} \,\Big|\, X_t=x \Big].$$
The same tower argument as before, with one extra factor of \(e^{-q\,\mathrm ds/\lambda}\approx 1 - q\,\mathrm ds/\lambda\) peeled off, gives
$$\partial_t \psi + \mathcal L\psi \;=\; \tfrac{q}{\lambda}\,\psi,$$
a backward equation with a killing term: paths pay rent \(q\) for the territory they occupy, and the expectation discounts them accordingly. Read in the other direction, this is the miracle the formula is famous for: the linear PDE on the left is solved by the path average on the right. Kac proved it after hearing Feynman lecture on path integrals; the imaginary-time Schrödinger equation is exactly this, with \(q\) the potential.
File the exact shape of this expression away. An exponential of a negated, accumulated cost, averaged over paths: we are going to meet it again as the solution of a control problem, and the whole essay turns on that meeting.
Now we stop spectating. Give the particle a motor: a control \(u_t\) we choose, added to the drift,
$$\mathrm dX_t \;=\; \big(b(X_t) + u_t\big)\,\mathrm dt \;+\; \sqrt{2T}\,\mathrm dW_t.$$
The noise stays; we do not get to switch off the weather, only to steer through it. And steering is priced. We fix a horizon \(T_{\!f}\), a terminal cost \(\Phi(x)\) saying where we would like to end up (low at the target), a running cost \(q(x)\ge0\) saying where we would rather not linger, and a quadratic price on effort. The problem is to choose the control policy minimizing the expected total,
$$C(x,t;u) \;=\; \mathbb E\Big[\, \Phi(X_{T_f}) \;+\; \int_t^{T_f}\Big( q(X_s) + \tfrac{\alpha}{2}\,|u_s|^2 \Big)\mathrm ds \,\Big|\, X_t = x\Big],$$
and the central object is the value function \(V(x,t) = \min_u C(x,t;u)\): the cost of the future, assuming we play it perfectly from here. (The clash of symbols with the landscape \(V\) of the previous essay is a century-old tradition; context will keep them apart, and the landscape mostly appears as the drift \(b=-\nabla V\).) The value function satisfies a PDE, by the same stand-at-the-first-instant argument that gave us the backward equation, upgraded with a decision: says that acting optimally over \([t,T_f]\) means acting optimally for one instant and then acting optimally from wherever that leaves us. Carrying out the expansion yields the :
For any small \(\mathrm ds\),
$$V(x,t) \;=\; \min_{u}\;\mathbb E_x\Big[\big(q(x)+\tfrac{\alpha}{2}|u|^2\big)\mathrm ds \;+\; V\big(X_{t+\mathrm ds},\,t+\mathrm ds\big)\Big],$$
where the minimum is over the action taken during the instant. The claim is that no strategy can beat "pay for one step, then be optimal": if a better full strategy existed, its restriction past \(t+\mathrm ds\) would beat the optimal continuation, a contradiction. This innocuous-looking recursion is the entire method; every Bellman equation in reinforcement learning is a discrete-time instance of it.
Its power has a famous price, named by Bellman himself: to use the recursion one must know \(V\) everywhere the next state might land, so the state space must be represented in full. In dimension one, a grid of 240 cells; in dimension fifty, more cells than atoms. He called it the curse of dimensionality, and it is why exact dynamic programming is a 1D-and-toy-problems luxury, and why the reformulation of Part IV, which replaces the grid with sampled paths, is more than an aesthetic preference.
Expand \(\mathbb E_x[V(X_{t+\mathrm ds}, t+\mathrm ds)]\) to first order. The time argument gives \(\partial_t V\,\mathrm ds\). The space argument moves under the controlled generator, drift \(b+u\), so it gives \((b+u)\cdot\nabla V\,\mathrm ds + T\Delta V\,\mathrm ds\). Insert, cancel \(V(x,t)\), divide by \(\mathrm ds\):
$$-\partial_t V \;=\; \min_u\Big\{\, q + \tfrac{\alpha}{2}|u|^2 + (b+u)\cdot\nabla V \,\Big\} + T\,\Delta V .$$
The bracket is quadratic in \(u\), so the minimum is explicit: \(u^{*} = -\nabla V/\alpha\), the control pointing straight down the value landscape, harder when the slope is steeper and cheaper effort permits. Substituting it back turns \(\min_u\{\cdot\}\) into \(q + b\cdot\nabla V - \tfrac{1}{2\alpha}|\nabla V|^2\) and yields the boxed equation.
One honesty flag: value functions of degenerate or constrained problems can fail to be differentiable, and the modern meaning of "solves HJB" is the theory of viscosity solutions (Crandall–Lions), which makes everything above rigorous without smoothness. Our \(T>0\) diffusion smooths \(V\) enough that we will not need it, but the term is worth knowing: it is where the subject's analytic depth lives.
$$\boxed{\;-\,\frac{\partial V}{\partial t} \;=\; q \;+\; b\cdot\nabla V \;+\; T\,\Delta V \;-\; \frac{1}{2\alpha}\,\big|\nabla V\big|^{2}\;}\qquad V(x,T_f)=\Phi(x),$$
with the optimal control read off as \(u^{*} = -\nabla V/\alpha\): descend the value function, exactly as a passive particle descends the energy landscape. Compare this equation to the backward equation of Part I, term by term. Three of the four terms on the right are literally \(\mathcal LV + q\): the backward operator, plus rent. Wanting has added exactly one term, \(-|\nabla V|^2/2\alpha\), and that term ruins everything. It is nonlinear, so solutions do not superpose, Monte Carlo path averages no longer represent them, and the clean duality of Figure 1 is gone. One quadratic term separates expecting from wanting, and it separates a nineteenth-century linear theory from a twentieth-century nonlinear one.
Unless it can be made to disappear.
Here is the observation the essay is named after. In the previous essay, exponentials of things kept turning nonlinear objects into linear ones: the Boltzmann distribution \(e^{-V/T}\) solved a flux balance, and \(F=-T\log Z\) turned products into sums. Try the same move on the value function. Pick a constant \(\lambda>0\), to be fixed shortly, and substitute
$$V(x,t) \;=\; -\,\lambda\,\log\psi(x,t)$$
into the HJB equation. , two second-order terms appear with \(|\nabla\psi|^2/\psi^2\) in them: one from the diffusion acting on the logarithm, one from the nonlinearity. They have opposite signs, and they cancel identically if and only if
From \(V=-\lambda\log\psi\): \(\;\partial_t V = -\lambda\,\partial_t\psi/\psi\), \(\;\nabla V = -\lambda\,\nabla\psi/\psi\), and
$$\Delta V \;=\; -\lambda\,\frac{\Delta\psi}{\psi} \;+\; \lambda\,\frac{|\nabla\psi|^2}{\psi^2}.$$
Substitute into \(-\partial_t V = q + b\cdot\nabla V + T\Delta V - \frac{1}{2\alpha}|\nabla V|^2\):
$$\lambda\frac{\partial_t\psi}{\psi} \;=\; q \;-\;\lambda\, b\cdot\frac{\nabla\psi}{\psi} \;-\;\lambda T\frac{\Delta\psi}{\psi} \;+\;\underbrace{\lambda T\,\frac{|\nabla\psi|^2}{\psi^2} \;-\; \frac{\lambda^2}{2\alpha}\frac{|\nabla\psi|^2}{\psi^2}}_{\text{cancels iff } \lambda = 2\alpha T}.$$
With \(\lambda=2\alpha T\), multiply through by \(-\psi/\lambda\):
$$\partial_t\psi \;+\; b\cdot\nabla\psi \;+\; T\Delta\psi \;=\; \frac{q}{\lambda}\,\psi,$$
which is \(\partial_t\psi + \mathcal L\psi = \frac{q}{\lambda}\psi\): the Feynman–Kac equation of Part I, verbatim, for the uncontrolled generator. The substitution is due to Hopf and Cole (who used it on Burgers' equation, which is what HJB becomes in one dimension with \(q=0\)); its use in control runs through Fleming, Holland, and, in the form presented here, Kappen's path-integral control and Todorov's linearly solvable MDPs.
$$\lambda \;=\; 2\,\alpha\, T .$$
Pause on what this condition says before enjoying what it buys. \(\alpha\) is the price of effort; \(T\) is the violence of the noise; the transform exists when the temperature of your uncertainty and the cost of your actions are locked to one another, channel by channel. In several dimensions the condition reads \(\alpha^{-1} \propto \Sigma\): you can only be cheaply clever in the directions nature is already noisy in. Where the wind blows, steering is affordable; in directions the noise never explores, control is priced out of the theory. This is the fine print under everything that follows, and we will return to it in the coda. Grant it, and the reward is total: with \(\lambda = 2\alpha T\), the HJB equation becomes, exactly,
$$\frac{\partial \psi}{\partial t} \;+\; \mathcal L\,\psi \;=\; \frac{q}{\lambda}\,\psi,\qquad \psi(x,T_f) = e^{-\Phi(x)/\lambda},$$
which is the Feynman–Kac equation from Part I, for the passive dynamics, with the costs demoted to a killing rate. The nonlinearity is gone. The equation of wanting was the equation of expecting all along, wearing a logarithm. The function \(\psi\), called the desirability, is a plain conditional expectation over uncontrolled paths, and the optimal control comes out as
$$u^{*}(x,t) \;=\; -\frac{\nabla V}{\alpha} \;=\; 2T\,\nabla \log \psi(x,t).$$
Look at that formula. In the previous essay, \(2T\)-scaled gradients of log-densities were the score, the velocity field along which diffusion transports probability, the thing generative models learn. Here the identical object reappears with a new meaning: the optimal action is the score of the desirability. Acting well is flowing up the gradient of the log-probability of a good future. The figure below solves the linear equation and lets the resulting controller drive.
The target is the slot on the right; paint penalty zones anywhere and the linear equation re-solves instantly. The blue value function rolls in backward from the horizon, the arrows are its downhill direction \(u^*=2T\nabla\log\psi\), and the green crowd is driven by them while the grey crowd drifts. Lower \(\lambda\) to make effort cheap and the steering turns vicious.
Two things are worth trying. Paint a wall of running cost between the start and the target and watch the value function route the crowd around it, discovering a detour nobody programmed. Then slide \(\lambda\). Since \(\lambda=2\alpha T\) at fixed noise, sliding it is repricing effort: small \(\lambda\) buys sharp, nearly deterministic steering toward the exact slot; large \(\lambda\) buys a lazy controller content to nudge. The slider will matter again in a moment, because \(\lambda\) is about to become a temperature in the statistical-mechanics sense, and "lazy versus sharp" is about to become "hot versus cold".
The transform did more than linearize a PDE. Feynman–Kac says the linear equation's solution is a path average, so we can write the desirability with no PDE at all:
$$\psi(x,t) \;=\; \mathbb E_{\text{passive}}\Big[\, \exp\Big(\!-\tfrac{1}{\lambda}\Big[\Phi(X_{T_f}) + \!\int_t^{T_f}\! q(X_s)\,\mathrm ds\Big]\Big) \,\Big|\, X_t = x\Big],$$
an average over uncontrolled futures, each weighted by the exponential of its own badness. And the corresponding statement upstairs, at the level of whole trajectories, is the theorem that gives this essay its name in practice: the optimally controlled process is the passive process, reweighted. Precisely, the probability the optimal controller assigns to a path \(\tau\) is
$$p^{*}(\tau) \;=\; \frac{1}{Z}\; p_{\text{passive}}(\tau)\; e^{-\,\mathrm{cost}(\tau)/\lambda},$$
a , with the path cost as energy and \(\lambda\) as temperature. Planning, in this class of problems, is therefore an algorithm anyone can implement in ten lines: dream a bundle of ordinary, uncontrolled futures; grade each one; keep them all, weighted by \(e^{-\mathrm{cost}/\lambda}\); act as the weighted ensemble does. No equation is solved. The figure does exactly this, next to the exact answer.
Girsanov's theorem prices a change of drift: steering with \(u\) instead of drifting costs, in log-likelihood, \(\frac{1}{4T}\int |u|^2\,\mathrm dt\) (heuristically: each instant, a Gaussian increment of variance \(2T\,\mathrm dt\) must be displaced by \(u\,\mathrm dt\)). So for any controlled path law \(p_u\),
$$\mathrm{KL}\big(p_u \,\|\, p_{\text{passive}}\big) \;=\; \mathbb E_{p_u}\Big[\tfrac{1}{4T}\!\int |u|^2\,\mathrm dt\Big] \;=\; \frac{1}{\lambda}\,\mathbb E_{p_u}\Big[\tfrac{\alpha}{2}\!\int |u|^2\,\mathrm dt\Big],$$
using \(\lambda=2\alpha T\): the control-effort cost is \(\lambda\) times a KL divergence. The whole objective becomes
$$\mathbb E_{p_u}\big[\Phi + \textstyle\int q\big] \;+\; \lambda\,\mathrm{KL}\big(p_u\|p_{\text{passive}}\big),$$
an expected state-cost plus an entropic penalty for disagreeing with nature. Minimizing an expression of the form \(\mathbb E_p[\mathcal E] + \lambda\,\mathrm{KL}(p\|p_0)\) over all distributions is a problem the previous essay solved twice: the minimizer is the tilted distribution \(p^* \propto p_0\,e^{-\mathcal E/\lambda}\), and the minimal value is a free energy, \(-\lambda\log \mathbb E_{p_0}[e^{-\mathcal E/\lambda}]\). Applied to path space, that minimizer is the boxed reweighting, and that free energy is exactly \(V = -\lambda\log\psi\). The value function of optimal control is the Helmholtz free energy of a Boltzmann distribution over futures. The hero equations of the two essays are the same equation.
Three hundred passive rollouts through the double well, each drawn with opacity proportional to its Boltzmann weight \(e^{-\mathrm{cost}/\lambda}\). Most dreams are worthless and vanish; the few that cross the barrier carry the plan. The dashed green mean of the surviving dreams lies on top of the blue exact answer, and the ESS readout counts how many dreams are actually doing the work.
Watch the as you cool \(\lambda\). At high temperature every dreamed future contributes and the plan is mush. At low temperature the plan sharpens, but the weight concentrates onto a handful of paths, then onto one, and the estimate is balanced on a single lucky rollout. This is the practical Achilles' heel of the whole path-integral method, visible here as fading spaghetti: the variance of importance sampling explodes exactly when you most want precision. It is also, from the right angle, the rare-event problem of the previous essay's Kramers section, since the good futures are barrier crossings, and those are exponentially rare dreams.
With normalized weights \(w_i\), \(\mathrm{ESS} = 1/\sum_i w_i^2\): it equals \(K\) when all \(K\) weights are uniform and \(1\) when one weight is everything. It is the standard diagnostic of importance sampling, and its collapse at low \(\lambda\) is not a numerical accident: the weights are \(e^{-\mathrm{cost}/\lambda}\), so their variance grows exponentially as \(\lambda\to 0\). Practical systems fight this with receding horizons (replan every few steps so errors cannot compound), by sampling around the previous plan instead of the passive dynamics, and by annealing \(\lambda\). All three tricks are visible in the MPPI literature, and all three are importance-sampling hygiene, which is the correct way to think about them.
Everything is now on the table for the slogan to become a theorem. Here is the inference problem. Take the passive dynamics as a prior over trajectories. Introduce a binary random variable \(\mathcal O\), read "things went well", whose likelihood given a trajectory is
$$p(\mathcal O = 1 \mid \tau) \;=\; \exp\big(-\,\mathrm{cost}(\tau)/\lambda\big),$$
so that cheap trajectories are likely to have gone well. Now do Bayes on path space: the posterior over trajectories given that things went well is prior times likelihood over evidence, which is
$$p(\tau \mid \mathcal O=1) \;=\; \frac{p_{\text{passive}}(\tau)\, e^{-\mathrm{cost}(\tau)/\lambda}}{Z} \;=\; p^{*}(\tau),$$
the optimal control law of Part IV, symbol for symbol. The controller of Figure 2 is not like a posterior; it is one. Its evidence \(Z\) is \(\psi\), its negative log-evidence is the value function, and the optimal action \(2T\nabla\log\psi\) is the score of the evidence. Every element of the control problem has an inference name, and the dictionary is exact:
passive dynamics = prior · cost = negative log-likelihood · desirability = evidence · value = negative log-evidence = free energy · optimal control = score of the evidence · temperature \(\lambda\) = how loosely "went well" is defined
There is one more face of the same theorem, and it is the most vivid one. Conditioning a Markov process on future information has a classical name, the , and its content is that the conditioned process is again a Markov diffusion, with the drift modified by \(2T\nabla\log h\), where \(h\) is the probability of the conditioning event. Set the event to "\(\mathcal O=1\)" and \(h\) is \(\psi\): the optimally controlled process and the conditioned process are the same process. Steering perfectly toward a goal is indistinguishable, in law, from being a passive particle that happens to be told the goal was reached. The figure tests this the hard way, by rejection.
Let \(h(x,t)\) be the probability (or density) of some future event, given \(X_t = x\); as a conditional expectation of the future it solves the backward equation, \(\partial_t h + \mathcal Lh = 0\). Bayes' rule at the level of transition kernels says the conditioned process moves as
$$p^{h}(x\to y) \;\propto\; p(x\to y)\,\frac{h(y, t+\mathrm dt)}{h(x,t)},$$
and expanding this tilt for a diffusion over a short step shifts the mean of the Gaussian increment by \(2T\nabla\log h\,\mathrm dt\): drift picks up exactly the score of the event probability. Brownian bridges are the textbook case (\(h\) = the heat kernel pinned at the endpoint), and the Schrödinger bridges of the previous essay's coda are the two-sided version. The identity "optimal control = \(h\)-transform with \(h=\psi\)" is why classifier guidance works in diffusion models: adding \(\nabla\log p(\text{label}\,|\,x)\) to a learned score is performing an \(h\)-transform on the generative diffusion, which by this essay is optimally steering it, with the classifier as terminal cost. Guided generation is stochastic optimal control, executed at sampling time.
Blue: run thousands of passive particles, keep only those whose endpoint "went well", and histogram the survivors at time t. Green: the optimally controlled crowd of Figure 2, same t. The histograms coincide at every intermediate time, and the acceptance readout shows what the agreement costs: inference by rejection throws almost everything away, control produces the same law without discarding a single sample.
The dictionary keeps paying after the essay ends, and one entry deserves its own paragraph because half the audience uses it daily. Discretize the recursion of Bellman's principle under this framework and the hard minimum softens into a :
In a discrete-time, discrete-action version of this problem class, the value recursion comes out as
$$V(x) \;=\; -\lambda\,\log \sum_a \exp\!\Big(\!-\tfrac{1}{\lambda}\,Q(x,a)\Big),$$
with \(Q(x,a)\) the cost-to-go of taking \(a\) now and continuing well. This is the soft Bellman equation of maximum-entropy reinforcement learning (soft Q-learning, SAC). Read it with Part II of the previous essay in mind: the sum is a partition function over actions, \(V\) is \(-\lambda\log Z\), a free energy, and the optimal policy is the corresponding Boltzmann distribution, \(\pi^{*}(a|x)\propto e^{-Q(x,a)/\lambda}\), i.e. a softmax over the negated \(Q\)-values. As \(\lambda\to0\) the log-sum-exp hardens into a max and greedy control returns; at \(\lambda>0\) the agent keeps its options deliberately open, paying a little cost for a lot of entropy. Readers from game theory will recognize the same object under yet another name: the quantal response / logit equilibrium, where softmax-rational play is exactly Boltzmann play. One functional, five fields, one temperature dial.
every value function in maximum-entropy RL is a free energy, every softmax policy is a Boltzmann distribution, and the "temperature" hyperparameter of those algorithms is the same \(\lambda\) that has been on a slider in every figure above. The previous essay ended by observing that its two universal objects, the Fokker–Planck equation and the free energy, were one; this one ends the same way. The value function is a free energy over futures. Bellman backups compute partition functions. Wanting, done optimally in the presence of noise, is thermodynamics pointed at the future.
An honest sequel names its hypotheses as loudly as its theorems. Three pieces of fine print, in increasing order of subtlety.
The channel condition. Everything hinged on \(\lambda=2\alpha T\): noise and control entering through the same channel, at locked exchange rates. Steer a dimension the noise does not excite, or price actions non-quadratically, and the \(|\nabla V|^2\) term survives, HJB stays nonlinear, and one is back to viscosity solutions and grids. The linearly solvable problems are a rich and useful class, and they are a class, with a boundary.
The variance. Figure 3's ESS collapse is not a demo artifact; it is the method's tax. The exact reweighting theorem holds at every \(\lambda\), but estimating it from finitely many dreams degrades exponentially as \(\lambda\to0\), which is precisely the regime of sharp, confident control. Every deployed descendant of this idea is, at heart, a variance-reduction scheme wrapped around a Boltzmann average.
The optimism. The subtlest one. Notice what the transform actually computed: , and an exponential average is not an average. It weights lucky outcomes more than unlucky ones, so the "control as inference" agent is mildly risk-seeking: it plans as if the noise were slightly on its side, because it conditions on things having gone well rather than making them go well in expectation. In benign problems the difference is a rounding error; in problems where a rare disaster lurks, it is the whole problem. The exactness of the theorem and the correctness of the objective are different questions, and this essay has only proved the first.
Expand the log-exponential average in cumulants:
$$-\lambda\log \mathbb E\big[e^{-C/\lambda}\big] \;=\; \mathbb E[C] \;-\; \frac{1}{2\lambda}\,\mathrm{Var}(C) \;+\; \cdots$$
The leading correction subtracts variance: among plans with equal expected cost, the exponential criterion prefers the riskier one, since its lucky tail is exponentially rewarded. (Readers of the previous essay have seen this expansion before: it is the free energy again, \(F = E - TS\) in cumulant form, with variance playing entropy's role.) This is classical risk-sensitive control (Whittle), and it is the precise technical content of critiques of "everything is inference" framings, including parts of the active-inference literature: converting costs into log-likelihoods silently converts expectations into free energies, and free energies have opinions about risk. If one wants risk-neutral optimal control, the identification is an approximation, and the approximation is uncontrolled exactly where danger is rare and large.
And the road. The two-sided version of the \(h\)-transform, conditioning on both endpoints, is the Schrödinger bridge, which is entropic optimal transport, which closes the loop back to the geometry of the previous essay; the modern samplers that steer diffusion models between distributions live there. The receding-horizon version of Figure 3 drives quadrotors and off-road vehicles as MPPI. The discrete-time version is maximum-entropy RL. The \(h\)-transform at sampling time is classifier guidance. And the equation all of them secretly solve is the one this essay started with: a linear backward equation, the oldest and mildest object in the theory, asked politely about the future.
Praise, then, for the backward equation: the forward equation's quiet twin, which prices options, ends gambler's ruin, and, given one logarithm, absorbs the entire theory of optimal behavior under noise into the theory of conditional expectation. Prediction and volition, the two things a mind does about the future, and the mathematics insists they were one subject. Few dualities in mathematics carry this much philosophy while looking this much like bookkeeping.