Maximum A Posteriori Estimation (MAP)

MLE’s Optimality

For large number of training samples, the Cramér-Rao lower (see an explanation at (Ben Lambert, 2014)) bound shows that no consistent estimator has a lower mean squared error than the maximum likelihood estimator. For these reasons (consistency and efficiency), maximum likelihood is often considered the preferred estimator to use for machine learning.

Maximum A Posteriori (MAP)

While MLE offers a provably “best fit” estimator of the data, it doesn’t account for any prior beliefs we may have about the distribution parameters. In other words, the estimate only depends on the dataset. This makes MLE prone to overfitting, and incorporating a prior over the parameter in not only the proper Bayesian way to estimate the posterior, but can also lead to a better estimator (cue regularization).

The empirical risk is the average loss over the data points.

Starting from the Bayes posterior for the parameter $\theta$, and ignoring $p(X)$ since it is independent of $\theta$, MAP is taking $\mathrm{argmax}$ over the posterior, which results in:

$${\mathbf{\theta }}_{\mathrm{MAP}}=\underset{\mathbf{\theta }}{\mathrm{argmax}}p(\mathbf{\theta }\mid \mathbf{x})=\underset{\mathbf{\theta }}{\mathrm{argmax}}\log p(\mathbf{x}\mid \mathbf{\theta })+\log p(\mathbf{\theta }).$$

We recognize, above on the right hand side, $\log p(\mathbf{x}\mid \mathbf{\theta })$, i.e. the standard log likelihood term, and $\log p(\theta )$, corresponding to the prior distribution.

MAP and Regularization

Many regularized estimation strategies, such as maximum likelihood learning regularized with weight decay, can be interpreted as making the MAP approximation to Bayesian inference. This view applies when the regularization consists of adding an extra term to the objective function that corresponds to $\log p(\theta )$.

Not all regularization penalties correspond to MAP Bayesian inference. For example, some regularizer terms may not be the logarithm of a probability distribution. Other regularization terms depend on the data, which of course a prior probability distribution is not allowed to do.

As an example, consider a linear regression model with a Gaussian prior on the weights $\mathbf{w}$. If this prior is given by $\mathcal{N}\left(\mathbf{w};0,\frac{1}{\lambda }{I}^2\right)$, then the log-prior term is proportional to the familiar $\lambda {\mathbf{w}}^T\mathbf{w}$ weight decay penalty, plus a term that does not depend on $\mathbf{w}$ and does not affect the learning process. MAP Bayesian inference with a Gaussian prior on the weights thus corresponds to weight decay.¹

Prior

In the scenarios where Bayesian estimation is typically used, the prior begins as a relatively uniform or Gaussian distribution with high entropy, and the observation of the data usually causes the posterior to lose entropy and concentrate around a few highly likely values of the parameters.

The prior has an influence by shifting probability mass density towards regions of the parameter space that are preferred a priori. In practice, the prior often expresses a preference for models that are simpler or more smooth. Critics of the Bayesian approach identify the prior as a source of subjective human judgment impacting the predictions.

MLE is MAP with Uniform Prior

With uniform prior for the parameter (thus making all $\theta$ equally likely and $p(\theta )$ a constant, thus removing the $\log p(\mathbf{\theta })$ term from $\mathrm{argmax}$), the MAP estimate becomes equal to the MLE, ${\hat{\theta }}_{\mathrm{MAP}}={\hat{\theta }}_{\mathrm{MLE}}$.

Footnotes

1. See the stackoverflow answer at MAP estimation as regularisation of MLE ↩

Ben Lambert (Director). (2014, February 21). Maximum Likelihood - Cramer Rao Lower Bound Intuition. https://www.youtube.com/watch?v=i0JiSddCXMM