running-placebo-analysis

pymc-labs's avatarfrom pymc-labs

Performs placebo-in-time sensitivity analysis to validate causal claims. Use when checking model robustness, verifying lack of pre-intervention effects, or ensuring observed effects are not spurious.

⭐1.1kstars🔀89forks📁View on GitHub🕐Updated Jan 10, 2026
[Experiment Analysis]

When & Why to Use This Skill

This Claude skill automates placebo-in-time sensitivity analysis to rigorously validate causal claims in experimental data. It helps users ensure model robustness by verifying the absence of pre-intervention effects, effectively distinguishing genuine causal impacts from spurious correlations or underlying noise.

Use Cases

  • Validating Interrupted Time Series (ITS) models: Use placebo tests to ensure that observed shifts in data are truly caused by the intervention rather than pre-existing trends.
  • Robustness checks for Difference-in-Differences (DiD): Verify the parallel trends assumption by simulating interventions at earlier time points to confirm no significant effects are detected.
  • Evaluating Synthetic Control experiments: Assess the credibility of synthetic counterfactuals by running placebo-in-time trials on pre-treatment data slices.
  • Scientific Research Validation: Strengthening the statistical significance of causal findings in academic or market research by ruling out accidental discoveries and false positives.
namerunning-placebo-analysis
descriptionPerforms placebo-in-time sensitivity analysis to validate causal claims. Use when checking model robustness, verifying lack of pre-intervention effects, or ensuring observed effects are not spurious.

Running Placebo Analysis

Executes placebo-in-time sensitivity analysis to validate causal experiments.

Workflow

  1. Define Experiment Factory: Create a function that returns a fitted CausalPy experiment (e.g., ITS, DiD, SC) given a dataset and time boundaries.
  2. Configure Analysis: Initialize PlaceboAnalysis with the factory, dataset, intervention dates, and number of folds (cuts).
  3. Run Analysis: Execute .run() to fit models on pre-intervention data folds.
  4. Evaluate Results: Compare placebo effects (which should be null) to the actual intervention effect. Use histograms and hierarchical models to quantify the "status quo" distribution.

Key Concepts

  • Placebo-in-time: Simulating an intervention at a time when none occurred to check if the model falsely detects an effect.
  • Fold: A slice of pre-intervention data used to test a placebo period.
  • Factory Pattern: Decouples the placebo logic from the specific CausalPy experiment type.

References