This is the third essay in a sequence. The first was about the forward equation: given the present, what the world will do. The second was about the backward equation: given a goal, what we should do. Between them sits a question both essays quietly assumed away: in every figure so far, we could see the state. The density was on screen; the controller read its own position off the grid. No real agent gets that. A real agent gets a hidden world and a trickle of noisy measurements, and must maintain, at every instant, its best guess about what is actually out there.
That problem is called filtering, and it completes a triptych with a symmetry too clean to be a coincidence. Prediction, volition, perception: what will be, what to do, what to believe. The forward equation owned the first, the backward equation owned the second, and this essay's thesis is that perception is owned by both at once. The optimal belief about the state at time \(t\), given everything ever observed, factorizes into a forward object \(\alpha_t\) carrying the past and a backward object \(\beta_t\) carrying the future, multiplied pointwise. That is the equation above the title, and both of its factors are solved by machinery we have already built, down to the same discretized operators.
The essay also has a destination beyond the theorem. The phrase "free energy" has followed us through both previous essays, and there is a research program in theoretical neuroscience, active inference, claiming that perception and action in real brains are both descents of one such functional. By the end of this essay every ingredient of that claim will be an object we have personally simulated, which puts us in a position to do something the surrounding discourse rarely does: say precisely which part of it is a theorem, which part is a modeling choice, and which part is a known problem. As always, underlined terms expand, we write in "we", and the figures are the argument.
The setup, called a , needs two ingredients. A hidden state \(X_t\), which we take to be the Langevin diffusion of essay one, drifting and jittering in a landscape we cannot watch. And an observation channel: every so often, a measurement arrives,
The pair (hidden Markov dynamics, noisy observations of it) is the workhorse of half of applied science: it is the hidden Markov model of speech and genomics when the state is discrete, the state-space model of econometrics and neuroscience when continuous, the basis of GPS positioning, target tracking, robot localization (SLAM), data assimilation in weather forecasting, and the decoding step of brain-computer interfaces. The Apollo guidance computer ran one. In every case the logical situation is identical: the world has a state, the state has dynamics, the sensors are noisy, and the object of interest is the conditional distribution of the former given the latter.
Two independence assumptions define the class: the state is Markov, and each observation depends only on the state at its own time. Everything in this essay flows from these two, and the factorization in the title is their joint consequence.
$$Y_k \;=\; X_{t_k} + \sigma_{\text{obs}}\,\varepsilon_k, \qquad \varepsilon_k \sim \mathcal N(0,1),$$
a ping: the true position, corrupted. The object we must maintain is the filtering distribution \(\pi_t(x) = p(X_t = x \mid \text{all pings so far})\), the honest summary of what the data permits us to believe. And the algorithm that maintains it is a two-step loop so natural it feels less derived than remembered, alternating the two halves of :
Predict. Between pings, no information arrives, so belief evolves exactly as probability does: by the Fokker–Planck equation of essay one. If we believed \(\pi_{t}\) a moment ago, we believe \(e^{\Delta t\,\mathcal L^{*}}\pi_t\) now. Uncertainty diffuses; the belief cloud spreads and drifts precisely as an unwatched crowd would, because as far as we know, the state is an unwatched crowd of possibilities.
Correct. When a ping \(y\) arrives, Bayes' rule reweighs every hypothesis by its likelihood \(g(x) = p(y \mid x)\):
$$\pi_{t}^{+}(x) \;=\; \frac{g(x)\,\pi_{t}^{-}(x)}{\int g\,\pi_{t}^{-}\,\mathrm dx}.$$
A multiplication and a renormalization; nothing moves, hypotheses are only promoted and demoted in place. Note which geometry this is: pointwise reweighting was exactly the Fisher–Rao flow of essay one's three-metrics figure, the one that could never cross a barrier. Filtering alternates the two geometries of that figure: transport when blind, reweighting when told. Prediction is Wasserstein; correction is Fisher–Rao.
The denominator looks like bookkeeping. Hold on to it: Part II is entirely about that denominator.
Between pings, the belief obeys the Fokker–Planck equation, spreading and drifting like the crowd it secretly is. At a ping, the belief is multiplied by the likelihood and renormalized: hypotheses compatible with the measurement are promoted, the rest demoted, nothing moves. Transport, then reweighting; blindness, then sight. The figure runs this loop live, and you should spend a minute being the hidden state: grab the particle and drag it somewhere the belief does not expect, and watch how many pings it takes for the cloud to hunt you down.
The dark dot is the hidden truth, shown only because we are gods here; hide it and steer by the belief alone, or drag the dot to ambush the filter. Between pings the blue cloud diffuses (it is literally running essay one's equation); each ping snaps it toward the green likelihood. Crank the noise and watch belief become appropriately humble.
Two behaviors in this figure are worth naming, because they are the whole phenomenology of perception in miniature. First, the belief cloud breathes: it contracts at each ping and relaxes between them, equilibrating between the information coming in and the entropy production of the dynamics. Steady-state perceptual uncertainty is a balance of exactly the two forces essay one spent itself on. Second, when the hidden particle hops the barrier between wells, the belief lags, hesitates bimodally with mass in both hypotheses, then commits. That bimodal hesitation is the filter being correct: for a few pings, two stories fit the data, and the honest belief holds both.
The loop above is discrete pings; nature usually observes continuously. Send the interval between pings to zero (scaling the noise appropriately) and the loop becomes a differential equation for \(\pi_t\), the , which is the exact and final answer to the perception problem, and which has a flaw of a familiar shape: it is nonlinear in the belief. Where essay two's HJB carried the nonlinearity \(|\nabla V|^2\), the equation of belief carries products of the posterior with its own conditional means. And in both essays the cure was the same kind of move, so we should try it here: stop insisting on the honest object and evolve a convenient one. Define \(\alpha_t\), the unnormalized belief, by running the identical predict-correct loop and skipping the renormalization. The result, in the continuous limit, is the , and it is linear. The equation of believing was linear all along, wearing a normalizer.
With continuous observations \(\mathrm dY_t = h(X_t)\,\mathrm dt + \mathrm dB_t\), the normalized filter evolves as
$$\mathrm d\pi_t \;=\; \mathcal L^{*}\pi_t\,\mathrm dt \;+\; \pi_t\,\big(h - \hat h_t\big)\,\big(\mathrm dY_t - \hat h_t\,\mathrm dt\big), \qquad \hat h_t = \int h\,\pi_t\,\mathrm dx.$$
The first term is Fokker–Planck: prediction. The second is the continuous-time correction, and it is built from the innovation \(\mathrm dY - \hat h\,\mathrm dt\), the part of each measurement the filter did not already expect. A filter learns only from its own surprise; measurements it predicted perfectly pass through without effect. (Readers who want the neuroscience keyword: this is precisely the "prediction error" that predictive-coding theories place at the center of cortical computation, appearing here as a theorem about conditional expectations rather than a hypothesis about neurons.)
The nonlinearity is the \(\hat h_t\): the equation for the belief contains averages under the belief. Solutions do not superpose, and the machinery of the previous essays stalls, until the next popup.
Drop the normalization and the unnormalized density obeys
$$\mathrm d\alpha_t \;=\; \mathcal L^{*}\alpha_t\,\mathrm dt \;+\; \alpha_t\, h\,\mathrm dY_t,$$
with no conditional means anywhere: linear in \(\alpha\), driven by the raw data. The honest posterior is recovered at any moment by one division, \(\pi_t = \alpha_t / \int\alpha_t\). The structural parallel with essay two is exact and worth savoring: there, the log of the value function obeyed a linear equation; here, the belief times an evidence factor obeys one. In both cases the nonlinearity of the famous equation is a normalization in disguise, and in both cases the linear object is the one worth computing.
Linearity is also what makes the theory robust: it is why filters can be analyzed, why the Feynman–Kac representation applies to perception too (\(\alpha\) is a path-integral over hypotheses weighted by observation likelihood), and why the particle methods of Part IV converge.
So the normalizer was the price of nonlinearity. It is also, and this is the pivot of the essay, the most valuable number in the theory. The total mass that the renormalizations keep discarding multiplies out, over a whole episode, to
$$\int \alpha_t(x)\,\mathrm dx \;=\; p\big(Y_1, \dots, Y_k \,\big|\, \text{model}\big),$$
the : the probability the model assigns to everything actually observed. Figure 1 has been accumulating its logarithm in the corner readout all along. One number, streaming in real time, scoring the model against reality.
This is the trilogy's most persistent character, and it is worth lining up its three costumes. In essay one, \(Z=\int e^{-V/T}\) normalized the Boltzmann distribution and \(-T\log Z\) was the equilibrium free energy. In essay two, \(\psi = \mathbb E[e^{-\mathrm{cost}/\lambda}]\) normalized the posterior over futures and \(-\lambda\log\psi\) was the value function. Here, \(\int\alpha_t\) normalizes the posterior over hidden presents, and \(-\log\int\alpha_t\) is the surprise the data has inflicted on the model so far. In all three, the same division of labor: the normalized object answers "what, given this frame"; the normalizer answers "how good is the frame". Model comparison, structure learning, and change-point detection all live in the normalizer, which is why throwing it away, though harmless for tracking, is scientifically expensive.
And minimizing \(-\log(\text{evidence})\) over model parameters, when the exact evidence is intractable, is done by bounding it with a variational free energy: the ELBO of essay two's Part V, now applied to trajectories. Part V of this essay is built on that observation.
Filtering is real-time: the belief at \(t\) uses pings up to \(t\). But offline, with a whole recorded episode in hand, the question changes: what should we believe about the state at noon, given the entire day's data, including the afternoon? This is smoothing, and its answer is the title equation. By the of the model, past and future data are independent given the present state, so their contributions factorize pointwise:
Split the data at \(t\): \(Y_{\text{past}}\) and \(Y_{\text{future}}\). Bayes, then the conditional independence of the two blocks given \(X_t\) (which holds because the state is Markov and observations touch only their own instant):
$$p(x_t \mid Y_{\text{past}}, Y_{\text{future}}) \;\propto\; p(x_t \mid Y_{\text{past}})\; p(Y_{\text{future}} \mid x_t) \;=\; \frac{\alpha_t(x)}{\int\alpha_t}\;\beta_t(x).$$
The forward factor \(\alpha_t\) is the filter we already have. The backward factor \(\beta_t(x) = p(Y_{\text{future}} \mid X_t = x)\) is a conditional expectation of future quantities given the present state, which is to say: it solves the Kolmogorov backward equation of essay two, punctuated by multiplication with each future ping's likelihood as the backward sweep absorbs it. Forward-and-multiply for the past; backward-and-multiply for the future. In the discrete-state world this pair is the forward-backward algorithm of hidden Markov models and the sum-product algorithm on chains; \(\beta\) is also, in the language of essay two, an \(h\)-function, and smoothing is an \(h\)-transform by the future data.
The conserved pairing follows immediately: \(\int \alpha_t\,\beta_t\,\mathrm dx = p(\text{all data})\), the total evidence, which cannot depend on where we chose to split past from future. At each ping, the likelihood factor hops from the \(\beta\) ledger to the \(\alpha\) ledger; the product never feels it.
$$p\big(x_t \mid \text{all data}\big) \;\propto\; \underbrace{\alpha_t(x)}_{\text{what the past supports}} \;\times\; \underbrace{\beta_t(x)}_{\text{what the future demands}},$$
where \(\beta_t(x)\), the likelihood of all future pings given \(X_t = x\), solves the backward equation of essay two, ping by ping. Readers of that essay have seen this figure's skeleton before: its Figure 1 propagated a density forward and an expectation backward and displayed their invariant pairing. That was, we can now admit, smoothing with the serial numbers filed off. Here the pairing \(\int\alpha_t\beta_t\,\mathrm dx\) is the total evidence of the episode, constant in \(t\) to machine precision, even across the pings, where a likelihood factor jumps from one ledger to the other in the same instant. Scrub the figure and watch hindsight at work: at times where the filter hesitated bimodally, the backward factor, knowing how the story ends, quietly deletes the wrong hypothesis.
One recorded episode: hidden path, pings, then both sweeps. Blue is real-time belief; green is the future's testimony; black is their product, the verdict of hindsight. The readout is \(\log\int\alpha\beta\), the episode's total evidence: it holds still to thirteen decimals while everything else moves, including at the pings.
Everything so far solved the belief on a grid, an option available in one dimension and a fantasy beyond a few. What the field wanted, from the 1960s on, was a finite-dimensional filter: a handful of summary statistics whose updates absorb every observation exactly. One glorious case delivers. If the dynamics are linear and everything is Gaussian, the belief stays Gaussian forever, its mean and covariance obey closed ODEs, and the result is the , arguably the most consequential algorithm of the twentieth century, and the source of a duality that readers of essay two are contractually owed:
For \(\mathrm dX = AX\,\mathrm dt + \mathrm dW\), \(\mathrm dY = CX\,\mathrm dt + \mathrm dB\), the posterior is \(\mathcal N(\hat x_t, P_t)\) with
$$\mathrm d\hat x = A\hat x\,\mathrm dt + P C^{\top}(\mathrm dY - C\hat x\,\mathrm dt), \qquad \dot P = AP + PA^{\top} + Q - PC^{\top}CP.$$
The mean chases the innovation with gain \(PC^\top\); the covariance obeys a Riccati equation. And that Riccati equation is, under the substitution \(A \to A^\top\), \(C \to B^\top\), time reversed, the Riccati equation of the linear-quadratic regulator: the optimal estimator and the optimal controller are transposes of one another. Kalman noticed this in the founding papers, and the previous essay's thesis, control is inference, has this as its oldest and most concrete instance, discovered from the other shore. Estimation is control of a belief; control is estimation of a desire.
It runs, famously, everywhere: Apollo navigation, GPS receivers, every phone's sensor fusion, every autopilot. The reason is the theorem, and the reason the theorem is fragile is the next popup.
Outside that case, the news is a flat negative, and it is one of the cleanest impossibility results in applied mathematics. For generic nonlinear systems, : the belief is an irreducibly infinite-dimensional object, and no fixed set of statistics, however clever, can carry it. Perception cannot be summarized; it can only be approximated. This is not a complaint about our algorithms. It is a statement about the problem, and it is why every real perceiving system, silicon or biological, is running an approximation, and why the interesting question is which one.
Brockett, Clark, and Mitter observed in the late 1970s that a finite-dimensional filter would furnish a finite-dimensional realization of the Zakai equation, and that this requires the Lie algebra generated by the equation's two operators (the Fokker–Planck operator and multiplication by \(h\)) to be finite-dimensional. For the deceptively innocent cubic sensor, Brownian motion observed through \(h(x)=x^3\), Hazewinkel, Marcus, and Sussmann proved the algebra is infinite-dimensional and no finite-dimensional exact filter of any kind exists. Generic nonlinearities behave the same way.
The known exceptions form a thin aristocracy: the linear-Gaussian class, and the Beneš family, where the drift satisfies a special Riccati-type identity that collapses the algebra. Everything else approximates: extended and unscented Kalman filters (local Gaussian fits), ensemble methods (the meteorologists' workhorse), variational filters (Part V), and the particle filters below. The impossibility theorem is why this list exists.
The approximation with the deepest roots in this trilogy is the : represent the belief by a swarm of hypothetical worlds, let each one live forward under the dynamics (predict), reweight each by the likelihood of the actual ping (correct), and occasionally resample, cloning heavy hypotheses and culling dead ones. It is the predict-correct loop with the grid replaced by an ensemble, and every failure mode it has, we have already met.
The algorithm (Gordon, Salmond, and Smith's 1993 bootstrap filter) is four lines. Its theory is a homecoming for this trilogy: a particle filter is an interacting particle system, since resampling couples the particles through their common weight normalization, and Del Moral's convergence theory proceeds exactly as essay one's Part IV did, by propagation of chaos: as \(N\to\infty\), the empirical swarm converges to the true filter, the mean-field limit of perception. The same mathematics that turned a crowd of magnetically coupled particles into the McKean–Vlasov equation turns a crowd of hypotheses into the Zakai equation.
Resampling is the interesting move. Without it, weights degenerate: one hypothesis slowly absorbs all the mass and the swarm wastes itself simulating worlds the data has already executed. With it, the swarm reallocates computation to where the posterior lives, at the price of extra variance and a genealogical collapse (trace ancestries backward and they coalesce). The diagnostic for when to trigger it is the effective sample size, making, by our count, its third appearance in the trilogy as the recurring villain: it measured planning collapse in essay two's Figure 3, importance-weight collapse before that, and here it decides when the swarm must be reborn.
Each green bead is a hypothetical world; pings reweight them and the ESS readout counts the survivors. Switch resampling off and watch degeneracy: within a few pings one bead owns everything and the swarm is dead weight. Switch it back on and the swarm reincarnates onto the living hypotheses, hugging the exact blue curve.
The trilogy's accounts are now complete enough to audit a famous claim. When the exact belief is unaffordable, the principled fallback is : pick a tractable family of beliefs \(q\), and minimize over it the functional
Essay two's Part V derived, for static unknowns, the identity \(F[q] = -\log(\text{evidence}) + \mathrm{KL}(q\,\|\,\text{posterior})\). Apply it with the hidden trajectory as the unknown and the ping sequence as the data: minimizing the free energy of a belief over paths simultaneously tightens a bound on this essay's evidence \(\int\alpha\) and drags \(q\) toward this essay's smoothing posterior \(\alpha\beta\). Every object in the identity is now something we have computed exactly, in one dimension, in the figures above; variational filtering is what one does in dimension fifty, where the grid is gone and the bound is all there is. Kalman smoothers, variational autoencoders with temporal structure, and deep state-space models are all this one minimization with different families \(q\).
One honest note on the word "perception is". The exact posterior is what perception should be; the variational minimizer is what a bounded perceiver can afford. The gap between them, \(\mathrm{KL}(q^*\|\,p)\), is not rhetoric: it is a computable number, it was on screen as the "variational gap" in essay one's Figure 3, and any theory that identifies perception with free-energy minimization is a theory of that gap being tolerable.
$$F[q] \;=\; \underbrace{-\log p(\text{data})}_{\text{fixed by the world}} \;+\; \underbrace{\mathrm{KL}\big(q \,\big\|\, p(\text{states}\mid\text{data})\big)}_{\text{the perceiver's error}},$$
the variational free energy of essay two, now over trajectories. Minimizing it is simultaneously approximating this essay's smoothing posterior and bounding this essay's evidence. And here the pieces click into a shape readers may recognize. The free energy principle of active inference asserts that a self-maintaining agent does two things with one functional: it updates beliefs to lower \(F\) (perception), and it selects actions to lower \(F\) (action). We are, for once, positioned to audit the assertion clause by clause, because we have personally built both halves.
What is theorem. Perception as free-energy descent: established, and it is this essay. Exact filtering minimizes \(F\) to its floor; variational filtering minimizes it within a family; the equations are Parts I through III. Action as free-energy descent: established for the linearly solvable class, and it is essay two, whose entire content was that optimal control minimizes an expected cost plus \(\lambda\,\mathrm{KL}\) from passive dynamics, a free energy over futures. Two theorems, one functional \(F\), one temperature. That much of active inference is mathematics, and it is good mathematics.
What is a modeling choice. The theorems above make an agent that perceives accurately and reaches goals cheaply. They do not make it curious, and a famous thought experiment says why: an agent whose only mandate is to keep \(F\) low can satisfy it in a dark room, where nothing happens and every prediction comes true. Active inference patches this with the expected free energy, an action-scoring rule whose decomposition contains an : prefer actions whose observations are expected to reduce uncertainty. The patch works, and the figure below shows it working, but it is important to file it correctly: it is not derived from the two theorems, it is an additional preference, chosen because agents without it are inert in exactly the situations where information must be bought with movement.
For a candidate action \(a\), the expected free energy decomposes (up to conventions that vary across the literature) as
$$G(a) \;=\; \underbrace{\mathbb E\big[\text{cost of outcomes}\big]}_{\text{pragmatic value}} \;-\; \underbrace{\mathbb E\big[\mathrm{KL}(q_{\text{after obs}} \,\|\, q_{\text{before}})\big]}_{\text{epistemic value: expected information gain}},$$
so the agent trades goal progress against expected reduction of its own uncertainty, and an observation is valuable in proportion to how much it is expected to change one's mind. The second term is classical Bayesian experimental design (Lindley, 1956), and maximizing it alone gives infotaxis, the search strategy of moths tracking odor plumes. The figure's toggle switches the term on and off; the weight on it is a free parameter with no theorem behind it, which is exactly the point being made in the main text.
The known objection carried over from essay two also applies here and deserves its one-line reprise: converting costs into log-probabilities makes the exponential-average, risk-seeking objective of that essay's coda, so agents of this family are structurally mild optimists, and no amount of epistemic value changes that.
The figure is the cleanest demonstration of that filing we could build. A corridor, a prize behind one of two doors, prior odds 60/40, and a map in the middle of the corridor that names the winning door to anyone standing near it. The purely pragmatic agent, minimizing expected distance to the prize, does the locally sensible thing: it walks to the favorite door and collects its 60%. The agent with the epistemic term pays movement it did not pragmatically need, reads the map, and collects nearly everything. Same filter, same world, one added term; the toggle is the entire philosophy of exploration, reduced to a button.
Episodes run continuously; the tallies keep score per policy. With the epistemic term off, the agent heads straight for the favorite door and wins about as often as the prior allows. Switch it on and watch the detour appear: the map is worth nothing to the destination and everything to the belief.
The fine print, in three clauses. Exactness is one-dimensional. Every "exact" curve on this page lives on a grid; the impossibility theorem of Part IV says that in general dimension there is no exact anything, only a portfolio of approximations, each with a regime. The model is assumed. Filtering is conditional probability given the model: the dynamics, the noise, the observation channel. Feed a filter the wrong model and it converges, confidently, to well-calibrated nonsense; the evidence readout of Part II is the only alarm in the building, and learning the model from that alarm (system identification, the EM algorithm, structure learning) is its own literature. The audit stands. Perception as free-energy descent is theorem; action as free-energy descent is theorem in the linearly solvable class; the epistemic term is an imported preference; the risk-seeking tilt of exponentiated costs is inherited by the whole family. Anyone selling more than that is selling.
And the road points somewhere specific. This essay coupled the forward and backward equations pointwise, by multiplication, with the world indifferent to being believed about. The remaining possibility is the interesting one: couple them dynamically. Let the crowd's density enter each agent's cost, so essay one's forward equation feeds essay two's backward equation, whose optimal control feeds back into the drift of the forward equation, and close the loop with a consistency demand: the crowd everyone optimizes against must be the crowd that optimizing produces. That fixed point is a Nash equilibrium of a continuum of agents, the system is a mean field game, and it is the natural fourth member of this sequence.
Praise, then, for the filter: the only algorithm in this trilogy that runs, right now, in the phone in your pocket and the mid-flight corrections of spacecraft; the machine that holds two stories when the data permits two, and drops one the instant it doesn't; the pointwise handshake of the trilogy's two equations. Prediction looks forward, control looks backward, and perception is where the two lines of sight cross: at the present, which is the only place anything is ever actually known.