How MCMC Sampling Helps Approximate Complex Posterior Distributions

 

How MCMC Sampling Helps Approximate Complex Posterior Distributions

Definition:
Markov Chain Monte Carlo (MCMC) is a computational method used in Bayesian statistics to generate samples from a posterior distribution when it is difficult to compute analytically.

Need:
In Bayesian inference,

$$p(\theta|D)=\frac{p(D|\theta)p(\theta)}{p(D)}$$

the denominator $p(D)$ involves a high-dimensional integral that is often impossible to evaluate exactly. Hence the posterior cannot be obtained in closed form.

How MCMC Helps:

1. Constructs a Markov chain:
   

   MCMC creates a sequence of parameter values where each new value depends only on the current one.

2. Targets the posterior distribution:
   

   The transition rules are designed so that the long-run (stationary) distribution of the chain equals the desired posterior 
   $p(\theta|D)$

3. .Generates samples from the posterior:
   

   After many iterations (and discarding burn-in), the generated values behave like random samples from the posterior.

4. Approximates posterior quantities:
   Using these samples:

   • Posterior mean ≈ sample average

   • Posterior variance ≈ sample variance

   • Credible intervals from sample quantiles

5. Handles complex models:
   

   MCMC works even for nonlinear, high-dimensional, or analytically intractable Bayesian models.

Conclusion:
MCMC converts Bayesian inference from a difficult integration problem into a sampling problem, allowing accurate approximation of complex posterior distributions.