Skip to main content

Validate a Power Simulation

Treat a power run as a model requiring diagnostics.

Reproducibility

Repeat the same CPU run with the same seed and compare the CSVs. Record the package version, input hash, YAML, seed, and execution backends.

Monte Carlo precision

Use power_ci_lower and power_ci_upper. If the interval straddles the target power, increase simulations before making a boundary decision. The interval quantifies simulation-count uncertainty only; it does not include uncertainty about the DGP specification.

DGP plausibility

Review dgp_rank, dgp_explained_var, and dgp_factor_dynamics. Compare observed and simulated distributions outside GeoSC where the decision is material: unit means and variances, cross-unit correlations, autocorrelation, seasonal patterns, extremes, and zeros. The current DGP does not model calendar effects or heteroskedastic idiosyncratic noise explicitly.

Grid stability

Power should usually rise with larger injected effects in a well-resolved simulation, but finite Monte Carlo estimates can move up or down. Investigate large reversals, high failure rates, coarse placebo p-values, or sensitivity to rank, history window, seed, donor eligibility, and effect pattern.

Do not call the run validated merely because all rows are valid. That flag requires zero failed simulations. Failed rows retain a conditional rejection rate and completion bounds, but ordinary power and its interval are unavailable. Do not select a design from those conditional rates or interpret an all-failed row as zero power. Remove the retired power_failure_rate_threshold key and regenerate older CSVs; decision consumers require current schema 2.0.