Skip to content

Before-and-after comparison with a comparison group

Optional detour  about 7 min

In this optional exercise, you evaluate a policy intervention by contrasting a simple pre-post analysis against a quasi-experimental difference-in-differences design that incorporates unexposed control sites. This exercise demonstrates why unadjusted temporal comparisons are vulnerable to secular confounding.

Before you start: Cleaned data containing temporal dates, site identifiers, and an intervention introduced on a known date (Setup).

Background

When an outcome changes following a clinical or policy intervention, other background factors (such as seasonal illness waves, staffing turnover, or regional shifts) may have occurred simultaneously. A simple pre-post comparison at the intervention site cannot isolate the intervention effect from these concurrent trends.

Incorporating a comparison group of unaffected sites illustrates what would have happened in the absence of the intervention. A difference-in-differences estimator computes:

\[\Delta_{\text{DID}} = (\bar{Y}_{\text{Intervention, After}} - \bar{Y}_{\text{Intervention, Before}}) - (\bar{Y}_{\text{Control, After}} - \bar{Y}_{\text{Control, Before}})\]

This design relies on the parallel trends assumption: assuming that intervention and control sites would have experienced parallel trajectories had the policy not been introduced. Consequently, report findings as consistent with an intervention effect rather than definitive causal proof.

General methodology

Direct the agent to compute the pre-post changes for both target and control sites alongside closed-form confidence intervals, requesting pre-intervention monthly trends to evaluate the parallel trends assumption:

Using <clean data>, evaluate the proportion of visits classified as <outcome> before and after <intervention date> at <intervention site>. Perform the identical calculation for the unexposed comparison sites combined. Report cell counts, baseline and follow-up rates, the absolute change within each group, and the difference-in-differences point estimate with a 95% confidence interval. Calculate standard errors using closed-form analytical formulas based on cell counts, avoiding computationally expensive bootstrap resampling. Provide a table of monthly outcome proportions across each site during the pre-intervention period, and document key methodological assumptions.

Explicitly specifying closed-form analytical formulas ensures the script executes in seconds. If an analysis script runs unexpectedly long, interrupt execution to ensure the agent has not defaulted to computationally intensive resampling loops (Steering).

Workshop example · Clinical: clinic network

Start from: the cleaned checkpoint (how to restore)
Produces: nothing required; ask for out/saturday.md to keep the result

Prompt

Using data/clean/visits_clean.csv, compare the share of visits that are urgent before and after 2025-03-08 at Northside. Then do the same for the other two clinics combined. Report the counts, the change at each, and the difference between the changes with a 95% confidence interval. Use a normal-approximation interval computed directly from the four counts, with no bootstrap or simulation. Also show the monthly urgent share for each site in the months before the change, as a table, and state your assumptions.

Files used: data/clean/visits_clean.csv, clinic_directory.html (confirms Saturday hours began 2025-03-08).

Reference results

Group Period Urgent share Visits
Northside before 2025-03-08 28.0% 382
Northside after 14.1% 1,850
Other clinics before 24.8% 990
Other clinics after 20.5% 4,228

The difference in differences is about −9.6 percentage points (95% CI about −15 to −4).

What this shows. Looking only at Northside, urgent visits fell by about 14 points. The other clinics also fell by about 4 points over the same period (partly seasonal: urgent visits are more common in winter months, and the "before" period is mostly winter). The comparison-adjusted estimate is therefore smaller than the naive one. That gap is the reason for the comparison group.

Caution about the baseline. The "before" period is only about ten weeks, mostly winter. A good report mentions it.

Verification criteria

  • Explicit baseline and follow-up partitions: Temporal intervals and comparison site definitions match the evaluation protocol exactly.
  • Complete contingency counts: Raw numerator and denominator counts are displayed for all four cells of the difference-in-differences matrix.
  • Documented methodological assumptions: The summary explicitly discusses the parallel trends assumption and potential unmeasured confounders.

Common pitfalls

Symptom Underlying cause Resolution
Script execution stalls for minutes The agent defaulted to computationally intensive bootstrap resampling or complex hierarchical modeling Interrupt execution and mandate closed-form analytical standard errors
Seasonal patterns bias estimates The pre and post observation periods span discordant seasons (e.g., comparing winter to summer) Restrict comparisons to matched seasonal months, or incorporate explicit calendar month controls
Sparse pre-intervention baseline Policy intervention was introduced near the beginning of data collection Acknowledge that the brief baseline window widens estimation uncertainty

Concepts behind this

  • Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
  • Scoping work. Scope requests to sizes that can be independently verified. Require structured plans before executing complex multi-step pipelines. Go deeper →

Builds on: Clean the data with a script, and show the flow