How does Gibbs sampling work and when is it preferred?

Question

AcadiFi · Accepted Answer

Gibbs sampling is an MCMC method that cycles through parameters, sampling each from its full conditional distribution given all others. When full conditionals have known form, Gibbs is simple and has 100% acceptance (no rejection step).

Nikolina Vasilev at Skopje Portfolio Research estimates a Bayesian factor model with 4 factors, 80 assets, and 500 monthly observations. Parameters: factor loadings B (80x4), factor variances, idiosyncratic variances, factor values F (500x4).

Gibbs sweep:
1. Sample loadings B given F, data, priors → Normal conditional
2. Sample factor values F given B, data, priors → Normal conditional
3. Sample idiosyncratic variances given B, F, data → Inverse-gamma conditional
4. Repeat — each step exploits conjugacy

In 8,000 iterations (2,000 burn-in), she gets full posterior for every parameter.

Gibbs wins when:
- Model has conditional conjugacy (normal-inverse-gamma, beta-binomial, Dirichlet-multinomial)
- Posterior is high-dimensional but each conditional is tractable
- You want to exploit model structure (blocked Gibbs updates multiple parameters together)

Limitations:
- Slow mixing when parameters are highly correlated — chain moves coordinate-by-coordinate
- Requires full conditionals to be known distributions
- Trapped in local modes for multimodal posteriors

Collapsed Gibbs: integrate out some parameters analytically, sample only the rest. This often dramatically accelerates convergence. For mixture models and topic models (LDA), collapsed Gibbs is the standard.

When conditionals are non-standard, use Metropolis-within-Gibbs: Gibbs for easy conditionals, Metropolis-Hastings step for hard ones. Most production Bayesian software (JAGS, WinBUGS) uses this hybrid under the hood.

How does Gibbs sampling work and when is it preferred?

Master Level III with our CFA Course

Related Questions

Practice Questions