Interpreting Results

GeoSC produces a suite of statistical indicators to help determine the validity and magnitude of an experiment’s impact.

Main Statistical Outputs

  • status: Result contract status. ok means the core ATT, p-value, and uncertainty fields are available; partial means the estimate ran but one or more uncertainty fields are unavailable; failed means required estimator outputs are missing and the effect should not be interpreted.
  • att (Average Treatment Effect on the Treated): The average difference between the observed outcome and the synthetic counterfactual during the post-treatment period.
  • p_value: The probability of observing an effect as large as the ATT under the null hypothesis (calculated via placebo permutations).
  • ci_lower / ci_upper: The SparseSC placebo confidence interval for the unscaled ATT when available.
  • se: Backward-compatible approximate SE derived from the reported placebo confidence interval width. This is not an estimator-provided sampling standard error.
  • se_approx_from_ci: The same approximation exposed under an explicit name. se_available remains false unless a future estimator provides a true sampling SE.
  • significant: Boolean flag indicating if the p-value falls below the configured alpha threshold (usually 0.05) when significance_available is true. Failed results always report significant: false.
  • assumption_status: Summary of structured assumption checks. skipped means checks were not run, pass means implemented checks did not flag a problem, and fail or error means causal interpretation needs review.

Placebo p-values are finite-sample and donor-pool dependent. With few eligible donor units, p-values are coarse. Missing confidence intervals must not be interpreted as zero uncertainty. Check ci.unavailable_reason to distinguish disabled CI computation from missing or non-finite estimator outputs. Check p_value_placebo_mode before relying on reproducibility: sampled means SparseSC capped the placebo enumeration.

Operational Interpretation Gates

Use status, data_validation.json, and assumption_validation.json together. status: "ok" means the artifact contract completed; it does not certify donor comparability, absence of spillovers, or production safety. Advisory assumption failures can mark otherwise available estimates as partial; required assumption failures mark inference as failed.

Do not combine scaled diagnostics with unscaled ATT, or compare GeoSC ATT to revenue, platform-reported lift, or other measurement-source outputs, unless the estimands, units, and time windows have been harmonized upstream.

Business Metrics

GeoSC provides raw lift estimates. It does not output built-in ROI or business-impact metrics (e.g., incremental revenue, ROAS). Use the statistical outputs (att) multiplied by your unit volumes and margin data in downstream reporting tools to calculate ROI.