Bayesian inference without posteriors
Conventional wisdom is that Bayesian inference flows from a posterior distribution: Let \(Y\sim P\) for some \(P\) in a model \(\{ P_\theta: \theta\in\Theta \}\) with densities \(\{ p(\cdot | \theta) : \theta\in \Theta\}\). Upon observing \(Y=y\) the density \(p(y|\theta)\) at the observed value \(y\) is combined with a prior density \(\pi(\theta)\) to define the posterior density \(\pi(\theta|y) \propto \pi(\theta)\times p(y|\theta)\), from which posterior means, confidence intervals and other summaries of \(\pi(\theta|y)\) may be computed. This is what I would call Bayesian modeling.
I was in a seminar today where the speaker was discussing point process models for spatial data using an intractable likelihood (a likelihood functional without a finite representation), and he made an offhand statement along the lines of “Bayesian methods require a likelihood”. For a model with densities \(\{ p(\cdot | \theta) : \theta\in \Theta\}\), the likelihood is \(p(y|\theta)\) as a function of \(\theta\), which appears to be a necessary component of the posterior distribution, and thus of Bayesian modeling.
However, an alternative starting point for Bayesian inference is Bayesian decision theory, which does not directly involve calculation of a posterior distribution, and so does not require a likelihood function in the usual sense. In this framework, a decision rule \(\hat \delta: \mathcal Y \rightarrow D\) is a Bayes rule for a loss function \(L: \Theta\times D\rightarrow R^+\) if it minimizes the prior expected loss, or Bayes risk: \[ R(\delta) = E [ L(\theta, \delta(Y)) ]. \] This definition requires no likelihood function and no posterior calculation.
Now one might argue that a posterior distribution is generally part of the solution to the Bayesian decision problem. After all, in general we can write \[\begin{align*} E[ L(\theta,\delta(Y)) ] & = E [ E [ L (\theta, \delta(Y)) | Y ] ], \end{align*}\] which suggests that the Bayes rule is given by \[\begin{align} \hat\delta(y)= \arg \min_{d\in D} E[ L(\theta,d ) | Y=y ]. \end{align}\] This seems to indicate that a Bayesian decision problem is solved by Bayesian modeling, i.e., calculation of a posterior distribution. While this is broadly true, there are some exceptions. For example, suppose the loss does not depend on \(\theta\), and so \(L(\theta,d) = f(d)\) for some function \(f\). A Bayes rule satisfies \(\hat\delta(y) = \arg\min_{d\in D} f(d)\) and is constant if the minimizer of \(f\) is unique, provided a constant decision rule is allowed. Therefore, a unique Bayes rule is obtained without calculating any posterior distribution.
This scenario becomes less trivial if the set of allowable decision rules is restricted. As an example, let \(Y\sim N(\theta,\sigma^2)\) and suppose we want to construct a level-\(\alpha\) test of \(H:\theta=0\) using a test statistic of the form \(t(Y) = | Y + b |\) for some \(b\in \mathbb R\). We have a prior distribution \(\theta\sim N(\mu,\tau^2)\), and wish to use the level-\(\alpha\) test that maximizes the prior expected power. In terms of a decision problem, an allowable decision rule is of the form \[\delta(y) = 1\times ( |y+b | > c_b )\] where \(c_b\) satisfies \(\Phi( (c_b -b )/\sigma) - \Phi( -(c_b+b)/\sigma) = 1-\alpha\), and our loss is \(L(\theta,\delta(y)) =1 - \delta(y)\). To minimize the loss, we would like the decision rule \(\delta(y) = 1\), but this is not allowed as it does not correspond to a level-\(\alpha\) test. So in this case, the Bayes rule is not found by pointwise posterior risk minimization. Instead, it can be found by directly maximizing the prior expected power, or equivalently, by obtaining the most powerful level-\(\alpha\) test of \(H: Y\sim N( 0,\sigma^2)\) versus \(K: Y\sim N( \mu,\tau^2+ \sigma^2 )\). The optimal test statistic is \(|y + \tilde b |\) with \(\tilde b=\mu\tfrac{\sigma^2}{\tau^2}\), and the acceptance region of the test is \[ - c_{\tilde b} - \tilde b \leq y \leq c_{\tilde b} - \tilde b,\] where \(c_{\tilde b}\) satisfies \(\Phi( ( c_{\tilde b} -\tilde b )/\sigma) - \Phi( -( c_{\tilde b} + \tilde b)/\sigma) = 1-\alpha\). Hoff (2022) referred to this test as the “Frequentist assisted by Bayes” (FAB) test. It was used by Pratt (1963) and Yu and Hoff (2018) to construct confidence intervals that maintain constant \(1-\alpha\) coverage across all values of \(\theta\), while also minimizing prior expected interval width.
When the loss does depend on the parameter, posterior-less Bayes procedures can still arise if the class of allowable procedures is sufficiently constrained. For example, consider estimation of \(\beta\in \mathbb R^p\) in the model \(y \sim N_n(X\beta, \Sigma(\beta))\), where \(\Sigma\) is a known function of \(\beta\). If we restrict ourselves to linear unbiased estimators, then the unique Bayes estimator under squared error loss is \[ \hat\beta = (X^\top \bar \Sigma^{-1 } X)^{-1} X^\top \bar\Sigma^{-1} y,\] where \(\bar \Sigma = E [ \Sigma(\beta) ]\). This estimator is also the GLS estimator in the model \(y \sim N_n( X\beta, \bar\Sigma)\).