Bayesian Model Specification: Some problems related to model choice and calibration
Krnjajic, Milovan
Krnjajic, Milovan
Loading...
Repository DOI
Publication Date
2011
Type
Conference Paper
Downloads
Citation
Draper, David (2011) Bayesian Model Specification: Some problems related to model choice and calibration Proceedings of AMSA 2011, Novosibirsk, Russia
Abstract
In the development of Bayesian model specification for inference and prediction we focus on the conditional distributions p([theta],[beta]) and p(D[theta],[beta]), with data D and background assumptions [beta], and consider calibration (an assessment of how often we get the right answers) as an important integral step of the model development. We compare several predictive model-choice criteria and present related calibration results. In particular, we have implemented a simulation study to compare predictive model-choice criteria LS[cv] , a log-score based on cross-validation, LS[fs], a full-sample log score, with deviance information criterion, DIC. We show that for several classes of models DIC and LS[cv] are (strongly) negatively correlated; that LS[fs] has better small-sample model discrimination performance than either DIC, or LS[cv] ; we further demonstrate that when validating the model-choice results, a standard use of posterior predictive tail-area for hypothesis testing can be poorly calibrated and present a method for its proper calibration.
Funder
Publisher
Publisher DOI
Rights
Attribution-NonCommercial-NoDerivs 3.0 Ireland