visual-predictive-checks
Guidelines for visual predictive checks following Säilynoja et al. recommendations using ArviZ
When & Why to Use This Skill
This Claude skill provides expert guidelines for performing Visual Predictive Checks (VPC) to validate Bayesian models using ArviZ and CmdStanPy. It streamlines the model validation workflow by offering standardized methods for comparing simulated and observed data, ensuring statistical calibration and reliability across various data types including continuous, count, and survival data based on Säilynoja et al. recommendations.
Use Cases
- Validating Bayesian model calibration: Using PIT ECDF plots and simultaneous bands to detect systematic biases in posterior distributions.
- Model fit assessment for discrete data: Utilizing rootograms and histograms to evaluate how well a model captures dispersion and frequency in count data.
- Survival analysis validation: Implementing Kaplan-Meier style predictive checks to ensure survival models accurately reflect censored data patterns.
- Comparative statistical analysis: Combining multiple summary statistics (median, MAD, IQR) with ECDF plots to provide a holistic view of model performance and avoid overfitting using LOO-PIT.
| name | visual-predictive-checks |
|---|---|
| description | Guidelines for visual predictive checks following Säilynoja et al. recommendations using ArviZ |
Visual Predictive Checks
Use this skill when running prior or posterior predictive checks to validate Bayesian models. These checks compare simulated data from the model to observed data (or plausible ranges for prior predictive checks).
ArviZ Workflow
- Fit model with CmdStanPy, generating predictive quantities in
generated quantitiesblock (e.g.,y_repfor posterior predictive,y_prior_predfor prior predictive) - Convert to ArviZ DataTree using
arviz_base.from_cmdstanpy, specifying predictive groups - Create visual checks using
arviz_plotsfunctions
Visual Checks by Data Type
Continuous Data
- Distribution:
plot_ppc_distwithkind="ecdf"andkind="kde" - PIT ECDF:
plot_ppc_pit- shows calibration with simultaneous bands - Coverage:
plot_ppc_pit(coverage=True)- equal-tailed interval coverage - Summary statistics:
plot_ppc_tstatfor median, MAD, IQR combined withcombine_plots - LOO-PIT:
plot_loo_pit- avoids double-dipping by using leave-one-out
Count Data
- Rootogram:
plot_ppc_rootogram- emphasizes discreteness and dispersion - Histogram:
plot_ppc_dist(kind="hist") - PIT ECDF and coverage: Same as continuous
Binary/Categorical/Ordinal Data
- Calibration:
plot_ppc_pava- PAV-adjusted calibration curves - Intervals:
plot_ppc_interval- posterior predictive intervals with observed overlay - PIT ECDF and coverage: Same as continuous
Censored/Survival Data
- Survival curves:
plot_ppc_censored- Kaplan-Meier style PPC - PIT ECDF and coverage: Same as continuous
Key Principles
Use multiple complementary views rather than relying on a single plot. For example, for continuous outcomes, combine ECDF (shows full distribution) with PIT ECDF (shows calibration) and t-stat PPCs (shows specific features like central tendency and spread).
LOO-PIT is preferred over regular PIT for posterior checks as it approximates leave-one-out predictive distribution and avoids overfitting concerns.
Name plots descriptively: prior_predictive_ecdf.png, loo_pit_calibration.png, posterior_rootogram.png.