Model Selection

Model selection is an important tool to select one of several competing statistical models. Recently, to specific frameworks have been frequently used to achieve a trade-off between model fit and complexity. On the one hand, Bayesian model selection rests on the Bayes factor, the odds of the marginal likelihoods for each model (in a way, a generalized likelihood ratio). On the other hand, minimum description length (MDL) is a principle for data compression. Both methods account for the functional flexibility of models and take order constraints into account.  

Fisher Information Approximation for MDL

A specific minimum description length (MDL) measure (i.e., FIA) is prone to errors in small samples. We’ve proposed a simple way to compute a lower-bound sample size that ensures that this specific bias cannot occur. This lower-bound N can simply be computed with this Excel Sheet. For details, see:

  • Heck, D. W., Moshagen, M., & Erdfelder, E. (2014). Model selection by minimum description length: Lower-bound sample sizes for the Fisher information approximation. Journal of Mathematical Psychology, 60, 29–34. doi:10.1016/j.jmp.2014.06.002

MDL and the Bayes Factor  

MDL and Bayes factors are asymptotically identical under specific conditions. However, some qualitative differences exist in finite samples as shown in

  • Heck, D. W., Wagenmakers, E.-J., & Morey, R. D. (2015). Testing order constraints: Qualitative differences between Bayes factors and normalized maximum likelihood. Statistics & Probability Letters, 105, 157–162. doi:10.1016/j.spl.2015.06.014

Priors for Reparameterized Models

The Bayes factor strongly depends on the specific prior distributions on the parameters. This becomes especially relevant if models with substantively meaningful parameters are reparameterized. We discuss this issue in case of MPT models, which are often reparameterized to include order constraints, and show how to derive adjusted priors for the new model:

  • Heck, D. W., & Wagenmakers, E.-J. (2016). Adjusted priors for Bayes factors involving reparameterized order constraints. Journal of Mathematical Psychology, 73, 110–116. doi:10.1016/j.jmp.2016.05.004