Analyze an exposure and outcome, crude and adjusted
In this exercise, you direct the agent to conduct a statistical analysis contrasting unadjusted (crude) comparisons against multivariable regression models adjusted for confounding covariates. Observing how point estimates shift after controlling for baseline imbalances provides a foundational lesson in empirical analysis.
Before you start: A cohort file containing one record per individual, an exposure indicator, an outcome endpoint, and candidate covariates (Setup).
Background
A crude statistical comparison evaluates raw differences in outcome rates between exposed and unexposed groups. In observational research, crude comparisons are frequently biased by confounding: baseline imbalances across characteristics that influence both exposure likelihood and outcome risk. For example, if older patients are more likely to receive clinical outreach and independently experience higher re-encounter rates, age confounds the observed association.
A multivariable logistic regression models the relationship between exposure and outcome while holding candidate covariates (such as age, sex, and baseline severity) constant, estimating an adjusted odds ratio (aOR):
- OR = 1.0: Indicates no association.
- OR > 1.0: Indicates higher odds of the outcome among the exposed.
- 95% confidence interval: Quantifies estimation uncertainty; intervals spanning 1.0 indicate associations consistent with chance variation.
In observational studies, estimated odds ratios reflect statistical association rather than definitive causal effect, and summary write-ups must adhere to cautious scientific phrasing.
Covariate selection represents a substantive methodological decision that must be determined a priori based on clinical reasoning rather than automated stepwise significance hunting.
General methodology
Direct the agent to compute both unadjusted and multivariable adjusted estimates, mandating explicit sample accounting across both models:
Using <cohort file>, analyze the association between <exposure> and <outcome>. Report raw sample sizes, event counts, and incidence rates for both exposed and unexposed cohorts. Calculate the unadjusted crude odds ratio, followed by an adjusted odds ratio from a multivariable logistic regression controlling for <covariates>, each with 95% confidence intervals. Save the analysis script to <scripts/analyze.py> and formatted results to <out/results.md>. Specify model formulas, regression diagnostics, and total sample counts utilized in each model (which should match <cohort size>). Provide a concise narrative interpreting why the adjusted point estimate shifts relative to the crude estimate.
Workshop example · Clinical: clinic network
Start from: the cohort checkpoint (how to restore)
Produces: scripts/analyze_pilot.py, out/pilot_results.md
Prompt
Using out/cohort.csv, compare HbA1c recheck between patients reached by a call and patients not reached. Report counts and rates for each group, then the unadjusted odds ratio, then an odds ratio from a logistic regression adjusted for age, sex and index HbA1c, each with a 95% confidence interval. Write the script to scripts/analyze_pilot.py and the results to out/pilot_results.md. State the model, the number of patients in each model (it should be 251), and why the adjusted estimate differs from the crude one.
Files used: out/cohort.csv (251 patients).
Reference results
| Group | Patients | Rechecked | Rate |
|---|---|---|---|
| Not reached | 159 | 47 | 30% |
| Reached | 92 | 45 | 49% |
| Model | Odds ratio | 95% CI |
|---|---|---|
| Unadjusted | 2.28 | 1.34 to 3.88 |
| Adjusted for age, sex, index HbA1c | 1.80 | 1.03 to 3.15 |
What this shows. The unadjusted estimate is larger because older patients were more likely both to be called and to have a recheck. The adjusted estimate is closer to the effect built into the synthetic data (an odds ratio of about 1.7), and its interval is wider and just excludes 1. A good write-up says the adjusted estimate is the one to report, and notes that the interval is wide and the design is observational.
Hand check. The unadjusted odds ratio is (45/47) / (47/112) = 0.957 / 0.420 ≈ 2.28. If your agent's number does not match this, find out why before going on.
Your adjusted value may differ slightly if the agent codes sex or age differently; it should still be near 1.8 with an interval close to 1.0 to 3.2.
Verification criteria
- Sample size consistency: Exposed and unexposed subgroup totals strictly sum to the total cohort sample size.
- Crude estimate reconciliation: The reported crude odds ratio matches a manual 2x2 contingency table cross-product calculation (\([A \times D] / [B \times C]\)).
- Model sample retention: Both unadjusted and adjusted models reflect identical sample sizes, confirming no records were dropped due to missing covariate entries.
- Cautious scientific interpretation: Narrative summaries discuss observed statistical associations without making unverified causal claims.
Common pitfalls
| Symptom | Underlying cause | Resolution |
|---|---|---|
| Sample size drops in adjusted model | Missing values in covariate columns triggered automatic listwise deletion | Impute covariates or verify why records contain missing covariate fields |
| Highly divergent estimates between runs | Non-deterministic sampling or shifting input files | Fix input paths; standard logistic regression estimation is completely deterministic |
| Extreme odds ratios or vast confidence intervals | Quasi-complete separation: zero events within a specific covariate sub-stratum | Re-evaluate model parameterization, collapse sparse categorical bins, or report model instability |
Concepts behind this
- Verification. Language models can be confidently incorrect. Require empirical evidence, verify mathematical invariants, and use clean-context peer review to audit findings. Go deeper →
- Scoping work. Scope requests to sizes that can be independently verified. Require structured plans before executing complex multi-step pipelines. Go deeper →
Builds on: Build a study cohort with a flow table